这是一个现网问题
问题现象
客户页面记录展示不全，只展示了1000条，最新的不展示。
项目组一兄弟定位为，数据库查询到的记录被get请求返回时给截掉了（由于最大size限制）
走给我审核时，第一直觉告诉我定位的原因肯定不对，因为如果是由于get请求大小限制，不可能刚好截断保留整整1000条记录，而且如果是按字节大小截断，response应该会报错吧？
验证猜想
抱着怀疑的态度，在网上搜了一下GET返回时有什么限制，见如下链接
Limit on the length of the data that a webserver can return in response to a GET request
There are no limits on the amount of data returned on a HTTP response from Jetty.
You could stream data back to the client until shortly before the heat death of the universe.
Technically speaking, you can have a HTTP Response with no Content-Length
specified, which can be returned using either the Chunked Transfer-Encoding, or just a raw stream of bytes with a Connection: close
indicating when the data is complete (done being sent) by a close of the underlying connection. Both of which are essentially limit-less.
If you use a HTTP Response with Content-Length, be aware that Content-Length is, in practice, a 32-bit number, but more modern browsers support the 64-bit versions.
大体意思就是GET在response返回时没有大小限制，说明定位原因不对。
根因推导
- 不是GET返回截断的问题，且返回不全，肯定是后台截断了，后台能从哪里截断？初步判断为数据库查询只返回了1000条记录
- 进一步搜索日志，发现果然是数据库返回了1000条记录
- 接着查看代码，发现代码在查数据库时并未使用limit限制，进一步猜测调用的beego orm的
All(container interface{}, cols ...string) (int64, error)有默认过滤规则 - 网上搜索关于beego orm All方法的资料并查看源码，发现All方法果然有默认1000的限制返回数
All / Values / ValuesList / ValuesFlat 受到 Limit 的限制，默认最大行数为 1000
源码：
func (o *querySet) All(container interface{}, cols ...string) (int64, error) {
return o.orm.alias.DbBaser.ReadBatch(o.orm.db, o, o.mi, o.cond, container, o.orm.alias.TZ, cols)
}
func (d *dbBase) ReadBatch(q dbQuerier, qs *querySet, mi *modelInfo, cond *Condition, container interface{}, tz *time.Location, cols []string) (int64, error) {
...
limit := tables.getLimitSQL(mi, offset, rlimit)
...
query := fmt.Sprintf("%s %s FROM %s%s%s T0 %s%s%s%s%s", sqlSelect, sels, Q, mi.table, Q, join, where, groupBy, orderBy, limit)
...
}
func (t *dbTables) getLimitSQL(mi *modelInfo, offset int64, limit int64) (limits string) {
if limit == 0 {
limit = int64(DefaultRowsLimit)
}
}
DefaultRowsLimit = 1000
Reference：
https://stackoverflow.com/questions/16172524/limit-on-the-length-of-the-data-that-a-webserver-can-return-in-response-to-a-get
http://zoomq.qiniudn.com/ZQScrapBook/ZqFLOSS/data/20131230002213/index.html
网友评论