.. _searching:

=========
Searching
=========

Once you start storing more data in ``qalx`` it will become a requirement
to be able to search for it.

``qalx`` provides different methods for searching your data


.. _searching_mongo:

You can query data using a `mongoDB query <https://docs.mongodb.com/manual/tutorial/query-documents/>`_
by providing a JSON serializable `query` to the `find` method

.. code-block:: python

    >>> qalx = QalxSession()
    >>> qalx.item.find(query={"data.CODE": "B1"})

Permissions
-----------

All searching in ``qalx`` will only return entities that your permissions have access to.
Any bot you start will inherit you permissions and will only return entities that you have access to.


Basic Usage
-----------

You can search for your data by providing the `query` argument containing your
mongoDB query to any ``qalx`` request that retrieves data.
Typically though you will just want to search for items using the `find()`
method.

`find` will return a `QalxListEntity` by default. This is a normal python `dict` with two fields `query` which
gives information about the search performed and `data` which will contain the returned entities.


.. note::
    When doing a `find` query for either either a `Set` or a `Group` be aware that the Items/Sets within
    will not be unpacked. This is to keep response times as low as possible.

.. code-block:: python
    :caption: Mongo

    >>> qalx = QalxSession()
    >>> qalx.item.add(data={"CODE": "B1", "thickness": 5, "length": 15})
    >>> # Returns entities with `data` containing `{"CODE": "B1"}`
    >>> b1_items = qalx.item.find(query={"data.CODE": "B1"})
    >>> print(b1_items['data'][0]['data'])
    {"CODE": "B1", "thickness": 5, "length": 15}

If you know that there is only a single entity containing the data you
are querying by you can use the `find_one` method to get just that
entity.  This will raise a `QalxMultipleEntityReturned` if multiple entities
are found or a `QalxEntityNotFound` if no entities are found with your search
criteria

.. code-block:: python
    :caption: Mongo

    >>> qalx = QalxSession()

    >>> qalx.item.add(data={"CODE": "B1", "thickness": 5, "length": 15},)

    # Will return an instance of `entities.item.Item`
    >>> qalx.item.find_one(query={"data.CODE": "B1"})



If you have tags specified on a Qalx session then any query to `find` or `find_one` will
include those tags on the query.

.. code-block:: python

    >>> from pyqalx import QalxSession
    >>> qalx = QalxSession(project_name="my-engineering-project")
    >>> qalx.tags.add(name="project_code", value="DEF456")

    # This will only return data which is tagged with a project_name of "my-engineering-project"
    # and a project_code of "DEF456"
    >>> qalx.item.find()

The above example is equivalent to the following.

.. code-block:: python

    >>> from pyqalx import QalxSession
    >>> qalx = QalxSession()
    >>> qalx.item.find({"$and": [
    ...         {"tags": {"$elemMatch": {"name": "project_code", "value": "DEF456"}}},
    ...         {"tags": {"$elemMatch": {"name": "project_name", "value": "my-engineering-project"}}}]})

.. warning::

    Remember that tags are always lowercased and stripped of whitespace by ``qalx``.  Ensure
    that if you are querying tags directly that you provide them in the correct case.


Advanced Usage
--------------

By default, any request to `find` will return a list of paginated results.  The quantity and page of results can be changed as follows:

.. code-block:: python
    :caption: return 100 items per page rather than 25

    >>> qalx = QalxSession()
    >>> # Return 100 entities per page rather than the default 25
    >>> qalx.item.find(query={"data.shape": "square"},
    ...                limit=100)


.. code-block:: python
    :caption: skip the first 50 entities

    >>> qalx = QalxSession()
    >>> # Skips the first 50 entities.  I.E. if there are 100 entities then
    >>> # this will return entities 50-75
    >>> qalx.item.find(query={"data.shape": "square"},
    ...                skip=50)


.. code-block:: python
    :caption: get next page of results

    >>> qalx = QalxSession()
    >>> # Returns a QalxListEntity paginated result
    >>> resp = qalx.item.find(query={"data.shape": "square"})
    >>> # print the items
    >>> print(resp.data)
    >>> # print the query
    >>> print(resp.query)
    >>> # Get the next page of results - note that you must create the
    >>> # QalxListEntity yourself
    >>> next_page = QalxListEntity(qalx.rest_api.get(qalx.item.find(query={"data.bentest": 5}).query.next)[1], child=Item)
    >>> # Print page two of the data
    >>> print(next_page.data)


Nested Lookups
--------------

With Mongo you can use the dot notation lookup for nested lookups.

.. warning::

    It is important that the lookup paths you supply are correct.  The lookups that you need to provide
    may differ from the structure of an entity that you receive in a ``qalx`` response.  For more information
    on the necessary lookups see :ref:`available search fields <available_search_fields>`


.. code-block:: python
    :caption: Mongo

    >>> qalx = QalxSession()
    >>> qalx.item.add(data={"owner": {"name": "Fred Bloggs"}})

    >>> qalx.item.find(query={"data.owner.name": "Fred Bloggs"})

You can also search by even more complex nested data structures

.. code-block:: python
    :caption: Mongo

    >>> qalx = QalxSession()
    >>> qalx.item.add(data={'details': [5, {'dimensions': {'weight': {'kg': 20}}}]})

    >>> # Get items where the weight is 20
    >>> qalx.item.find(query={"data.details.dimensions.weight.kg": 20})
    >>> # Get items where the number 5 is in the details list
    >>> qalx.item.find(query={"data.details": 5})

Complex Searching
-----------------

``qalx`` enables you to do much more complex filtering than just ``X=Y``.

See `here <https://docs.mongodb.com/manual/reference/operator/query/>`_ for more
mongoDB operators.

``lte``
=======

.. code-block:: python
    :caption: Mongo

    >>> qalx = QalxSession()

    # Will return items with a weight <= 15
    >>> qalx.item.find(query={"data.weight": {"$lte": 15}})


``gte``
=======

.. code-block:: python
    :caption: Mongo

    >>> qalx = QalxSession()

    # Will return items with a weight >= 15
    >>> qalx.item.find(query={"data.weight": {"$gte": 15}})


``lt``
======

.. code-block:: python
    :caption: Mongo

    >>> qalx = QalxSession()

    # Will return items with a weight < 15
    >>> qalx.item.find(query={"data.weight": {"$lt": 15}})

``gt``
======

.. code-block:: python
    :caption: Mongo

    >>> qalx = QalxSession()

    # Will return items with a weight > 15
    >>> qalx.item.find(query={"data.weight": {"$gt": 15}})


``in``
======

.. code-block:: python
    :caption: Mongo

    >>> qalx = QalxSession()

    >>> qalx.item.add(data={'length': 12})
    >>> qalx.item.add(data={'length': 15})

    # Will return items with a weight of either 12 or 15
    >>> qalx.item.find(query={"data.weight": {"$in": [12, 15]}})

``and``
=======

To search using a logical ``and`` query provide a list of queries to filter against
rather than a single value

.. code-block:: python
    :caption: Mongo

    >>> qalx = QalxSession()

    # Will return items with a weight of 15 and a length of 12
    >>> qalx.item.find(query={"data.weight": 15, "length": 12})

``or``
=======

.. code-block:: python
    :caption: Mongo

    >>> qalx = QalxSession()

    # Will return items with a length of 15 or a weight of 12
    # Be aware that you need to specify the `data` key again when specifying `$or` queries
    >>> qalx.item.find(query={"$or": [{"data.weight": 15}, {"data.length": 12}]})


Datetime Searching
------------------

Searching via datetime fields can be done by supplying a datetime object.  This works for system stored datetimes as well as datetimes in your data.
Timezone aware and timezone naive datetimes are supported.

.. note::
    All system stored datetimes in ``qalx`` are stored as UTC.  Be aware of this
    if providing timezone aware datetimes when querying by system stored datetimes.

Querying by created date is most effective when combining with other search operators

.. code-block:: python
    :caption: Mongo

    >>> from datetime import datetime
    >>> date_from = datetime(2019, 5, 24, 0, 0, 0)
    >>> date_to = datetime(2019, 5, 30, 23, 59, 59)

    >>> qalx = QalxSession()
    >>> qalx.item.find(query={"created": {
    ...                          "$gte": date_from,
    ...                          "$lte": date_to,}
    ...                         })

Combining Search Operators
--------------------------

You can combine search operators in a highly flexible way

.. code-block:: python
    :caption: Mongo

    >>> from datetime import datetime
    >>> date_from = datetime(2019, 5, 24, 0, 0, 0)
    >>> date_to = datetime(2019, 5, 30, 23, 59, 59)

    >>> qalx = QalxSession()
    >>> qalx.item.find(query={"data.weight": {"$in": [14,15]},
    ...                       "data.length": {"$gte": 12},
    ...                       "data.time_run": {"$gte": datetime.now()},
    ...                       "$or": [{"data.thickness": 20}, {"data.height": 21}],
    ...                       "metadata.data.owner.name": "Fred Bloggs",
    ...                       "created": {
    ...                          "$gte": date_from,
    ...                          "$lte": date_to,}
    ...                       })

.. _available_search_fields:

Available Search Fields
-----------------------

The following field keyword arguments are available to search by for each entity

Item
====
* data
* meta
    * Mongo Syntax: `qalx.item.find(query={'metadata.data': <query>})`
* file
* tags
* created
* guid

Set
===
* meta
    * Mongo Syntax: `qalx.set.find(query={'metadata.data': <query>})`
* created
* guid

Group
=====
* meta
    * Mongo Syntax: `qalx.group.find(query={'metadata.data': <query>})`
* created
* guid

Bot
===
* config
* meta
    * Mongo Syntax: `qalx.bot.find(query={'metadata.data': <query>})`
* created
* guid
* status message
    * Mongo Syntax: `qalx.bot.find(query={'status.message': 'SOME STATUS'})`
* status modified
    * Mongo Syntax: `qalx.bot.find(query={'status.modified': <datetime string>})`

Queue
=====
* parameters
* meta
    * Mongo Syntax: `qalx.queue.find(query={'metadata.data': <query>})`
* created
* guid

Blueprint
=========
* schema
* entity_type
* meta
    * Mongo Syntax: `qalx.blueprint.find(query={'metadata.data': <query>})`
* created
* guid

Worker
======

It is not possible to query directly by a worker via the API.