美文网首页golang notes
beego-orm查询使用All查不到结果问题

beego-orm查询使用All查不到结果问题

作者: 风千寻艾 | 来源:发表于2017-10-19 11:32 被阅读0次

    这是一个现网问题

    问题现象

    客户页面记录展示不全,只展示了1000条,最新的不展示。

    项目组一兄弟定位为,数据库查询到的记录被get请求返回时给截掉了(由于最大size限制)
    走给我审核时,第一直觉告诉我定位的原因肯定不对,因为如果是由于get请求大小限制,不可能刚好截断保留整整1000条记录,而且如果是按字节大小截断,response应该会报错吧?

    验证猜想

    抱着怀疑的态度,在网上搜了一下GET返回时有什么限制,见如下链接

    Limit on the length of the data that a webserver can return in response to a GET request

    There are no limits on the amount of data returned on a HTTP response from Jetty.
    You could stream data back to the client until shortly before the heat death of the universe.
    Technically speaking, you can have a HTTP Response with no Content-Length
    specified, which can be returned using either the Chunked Transfer-Encoding, or just a raw stream of bytes with a Connection: close
    indicating when the data is complete (done being sent) by a close of the underlying connection. Both of which are essentially limit-less.
    If you use a HTTP Response with Content-Length, be aware that Content-Length is, in practice, a 32-bit number, but more modern browsers support the 64-bit versions.

    大体意思就是GET在response返回时没有大小限制,说明定位原因不对。

    根因推导
    1. 不是GET返回截断的问题,且返回不全,肯定是后台截断了,后台能从哪里截断?初步判断为数据库查询只返回了1000条记录
    2. 进一步搜索日志,发现果然是数据库返回了1000条记录
    3. 接着查看代码,发现代码在查数据库时并未使用limit限制,进一步猜测调用的beego orm的All(container interface{}, cols ...string) (int64, error)有默认过滤规则
    4. 网上搜索关于beego orm All方法的资料并查看源码,发现All方法果然有默认1000的限制返回数

    All / Values / ValuesList / ValuesFlat 受到 Limit 的限制,默认最大行数为 1000

    源码:

    func (o *querySet) All(container interface{}, cols ...string) (int64, error) {
        return o.orm.alias.DbBaser.ReadBatch(o.orm.db, o, o.mi, o.cond, container, o.orm.alias.TZ, cols)
    }
    
    func (d *dbBase) ReadBatch(q dbQuerier, qs *querySet, mi *modelInfo, cond *Condition, container interface{}, tz *time.Location, cols []string) (int64, error) {
    ...
        limit := tables.getLimitSQL(mi, offset, rlimit)
    ...
        query := fmt.Sprintf("%s %s FROM %s%s%s T0 %s%s%s%s%s", sqlSelect, sels, Q, mi.table, Q, join, where, groupBy, orderBy, limit)
    ...
    }
    
    func (t *dbTables) getLimitSQL(mi *modelInfo, offset int64, limit int64) (limits string) {
        if limit == 0 {
            limit = int64(DefaultRowsLimit)
        }
    }
    
    DefaultRowsLimit = 1000
    
    

    Reference:
    https://stackoverflow.com/questions/16172524/limit-on-the-length-of-the-data-that-a-webserver-can-return-in-response-to-a-get
    http://zoomq.qiniudn.com/ZQScrapBook/ZqFLOSS/data/20131230002213/index.html

    相关文章

      网友评论

        本文标题:beego-orm查询使用All查不到结果问题

        本文链接:https://www.haomeiwen.com/subject/qtyduxtx.html