Searching¶
Once you start storing more data in qalx it will become a requirement
to be able to search for it.
qalx provides different methods for searching your data
You can query data using a mongoDB query by providing a JSON serializable query to the find method
>>> qalx = QalxSession()
>>> qalx.item.find(query={"data.CODE": "B1"})
Permissions¶
All searching in qalx will only return entities that your permissions have access to.
Any bot you start will inherit you permissions and will only return entities that you have access to.
Basic Usage¶
You can search for your data by providing the query argument containing your
mongoDB query to any qalx request that retrieves data.
Typically though you will just want to search for items using the find()
method.
find will return a QalxListEntity by default. This is a normal python dict with two fields query which gives information about the search performed and data which will contain the returned entities.
Note
When doing a find query for either either a Set or a Group be aware that the Items/Sets within will not be unpacked. This is to keep response times as low as possible.
>>> qalx = QalxSession()
>>> qalx.item.add(data={"CODE": "B1", "thickness": 5, "length": 15})
>>> # Returns entities with `data` containing `{"CODE": "B1"}`
>>> b1_items = qalx.item.find(query={"data.CODE": "B1"})
>>> print(b1_items['data'][0]['data'])
{"CODE": "B1", "thickness": 5, "length": 15}
If you know that there is only a single entity containing the data you are querying by you can use the find_one method to get just that entity. This will raise a QalxMultipleEntityReturned if multiple entities are found or a QalxEntityNotFound if no entities are found with your search criteria
>>> qalx = QalxSession()
>>> qalx.item.add(data={"CODE": "B1", "thickness": 5, "length": 15},)
# Will return an instance of `entities.item.Item`
>>> qalx.item.find_one(query={"data.CODE": "B1"})
If you have tags specified on a Qalx session then any query to find or find_one will include those tags on the query.
>>> from pyqalx import QalxSession
>>> qalx = QalxSession(project_name="my-engineering-project")
>>> qalx.tags.add(name="project_code", value="DEF456")
# This will only return data which is tagged with a project_name of "my-engineering-project"
# and a project_code of "DEF456"
>>> qalx.item.find()
The above example is equivalent to the following.
>>> from pyqalx import QalxSession
>>> qalx = QalxSession()
>>> qalx.item.find({"$and": [
... {"tags": {"$elemMatch": {"name": "project_code", "value": "DEF456"}}},
... {"tags": {"$elemMatch": {"name": "project_name", "value": "my-engineering-project"}}}]})
Warning
Remember that tags are always lowercased and stripped of whitespace by qalx. Ensure
that if you are querying tags directly that you provide them in the correct case.
Advanced Usage¶
By default, any request to find will return a list of paginated results. The quantity and page of results can be changed as follows:
>>> qalx = QalxSession()
>>> # Return 100 entities per page rather than the default 25
>>> qalx.item.find(query={"data.shape": "square"},
... limit=100)
>>> qalx = QalxSession()
>>> # Skips the first 50 entities. I.E. if there are 100 entities then
>>> # this will return entities 50-75
>>> qalx.item.find(query={"data.shape": "square"},
... skip=50)
>>> qalx = QalxSession()
>>> # Returns a QalxListEntity paginated result
>>> resp = qalx.item.find(query={"data.shape": "square"})
>>> # print the items
>>> print(resp.data)
>>> # print the query
>>> print(resp.query)
>>> # Get the next page of results - note that you must create the
>>> # QalxListEntity yourself
>>> next_page = QalxListEntity(qalx.rest_api.get(qalx.item.find(query={"data.bentest": 5}).query.next)[1], child=Item)
>>> # Print page two of the data
>>> print(next_page.data)
Nested Lookups¶
With Mongo you can use the dot notation lookup for nested lookups.
Warning
It is important that the lookup paths you supply are correct. The lookups that you need to provide
may differ from the structure of an entity that you receive in a qalx response. For more information
on the necessary lookups see available search fields
>>> qalx = QalxSession()
>>> qalx.item.add(data={"owner": {"name": "Fred Bloggs"}})
>>> qalx.item.find(query={"data.owner.name": "Fred Bloggs"})
You can also search by even more complex nested data structures
>>> qalx = QalxSession()
>>> qalx.item.add(data={'details': [5, {'dimensions': {'weight': {'kg': 20}}}]})
>>> # Get items where the weight is 20
>>> qalx.item.find(query={"data.details.dimensions.weight.kg": 20})
>>> # Get items where the number 5 is in the details list
>>> qalx.item.find(query={"data.details": 5})
Complex Searching¶
qalx enables you to do much more complex filtering than just X=Y.
See here for more mongoDB operators.
lte¶
>>> qalx = QalxSession()
# Will return items with a weight <= 15
>>> qalx.item.find(query={"data.weight": {"$lte": 15}})
gte¶
>>> qalx = QalxSession()
# Will return items with a weight >= 15
>>> qalx.item.find(query={"data.weight": {"$gte": 15}})
lt¶
>>> qalx = QalxSession()
# Will return items with a weight < 15
>>> qalx.item.find(query={"data.weight": {"$lt": 15}})
gt¶
>>> qalx = QalxSession()
# Will return items with a weight > 15
>>> qalx.item.find(query={"data.weight": {"$gt": 15}})
in¶
>>> qalx = QalxSession()
>>> qalx.item.add(data={'length': 12})
>>> qalx.item.add(data={'length': 15})
# Will return items with a weight of either 12 or 15
>>> qalx.item.find(query={"data.weight": {"$in": [12, 15]}})
and¶
To search using a logical and query provide a list of queries to filter against
rather than a single value
>>> qalx = QalxSession()
# Will return items with a weight of 15 and a length of 12
>>> qalx.item.find(query={"data.weight": 15, "length": 12})
or¶
>>> qalx = QalxSession()
# Will return items with a length of 15 or a weight of 12
# Be aware that you need to specify the `data` key again when specifying `$or` queries
>>> qalx.item.find(query={"$or": [{"data.weight": 15}, {"data.length": 12}]})
Datetime Searching¶
Searching via datetime fields can be done by supplying a datetime object. This works for system stored datetimes as well as datetimes in your data. Timezone aware and timezone naive datetimes are supported.
Note
All system stored datetimes in qalx are stored as UTC. Be aware of this
if providing timezone aware datetimes when querying by system stored datetimes.
Querying by created date is most effective when combining with other search operators
>>> from datetime import datetime
>>> date_from = datetime(2019, 5, 24, 0, 0, 0)
>>> date_to = datetime(2019, 5, 30, 23, 59, 59)
>>> qalx = QalxSession()
>>> qalx.item.find(query={"created": {
... "$gte": date_from,
... "$lte": date_to,}
... })
Combining Search Operators¶
You can combine search operators in a highly flexible way
>>> from datetime import datetime
>>> date_from = datetime(2019, 5, 24, 0, 0, 0)
>>> date_to = datetime(2019, 5, 30, 23, 59, 59)
>>> qalx = QalxSession()
>>> qalx.item.find(query={"data.weight": {"$in": [14,15]},
... "data.length": {"$gte": 12},
... "data.time_run": {"$gte": datetime.now()},
... "$or": [{"data.thickness": 20}, {"data.height": 21}],
... "metadata.data.owner.name": "Fred Bloggs",
... "created": {
... "$gte": date_from,
... "$lte": date_to,}
... })
Available Search Fields¶
The following field keyword arguments are available to search by for each entity
Item¶
data
- meta
Mongo Syntax: qalx.item.find(query={‘metadata.data’: <query>})
file
tags
created
guid
Set¶
- meta
Mongo Syntax: qalx.set.find(query={‘metadata.data’: <query>})
created
guid
Group¶
- meta
Mongo Syntax: qalx.group.find(query={‘metadata.data’: <query>})
created
guid
Bot¶
config
- meta
Mongo Syntax: qalx.bot.find(query={‘metadata.data’: <query>})
created
guid
- status message
Mongo Syntax: qalx.bot.find(query={‘status.message’: ‘SOME STATUS’})
- status modified
Mongo Syntax: qalx.bot.find(query={‘status.modified’: <datetime string>})
Queue¶
parameters
- meta
Mongo Syntax: qalx.queue.find(query={‘metadata.data’: <query>})
created
guid
Blueprint¶
schema
entity_type
- meta
Mongo Syntax: qalx.blueprint.find(query={‘metadata.data’: <query>})
created
guid
Worker¶
It is not possible to query directly by a worker via the API.
