Defense.com's SIEM platform gives you live access to your log data from the last 90 days (with 12 months stored in total, including cold storage). This is designed to help you meet PCI DSS requirements for the retention of log data.
From time to time, you may need to review or query your logs to audit activity or understand what's happened at a particular time. That's where Lucene Query Language comes in.
Accessing the log search
From the Defense.com dashboard, head over to the navigation on the left-hand side and select SIEM
Then, from the expanded menu, select Log Search
You'll be presented with the log search page where you can view, query and expand the logs that have been ingested by our SIEM platform.
Filtering by time
By default, this page will display the logs we've received in the last 15 minutes. To widen your search, click on the Last 15 Minutes button under the Date Range heading. You'll then be able to select from some preset timeframes or select a custom to and from date/time.
If you click on any of the logs on this page, it will expand, and you'll be presented with a table view which includes all the fields and data within it. These will typically include things like host, port or type. These fields form the basis of most Lucene queries.
Not all log sources contain the same fields. Some may use host to denote a hostname, others may use hostname or something completely different. By expanding logs, you'll be able to see all the fields and their associated data.
Creating a query
Towards the top of the page, you should see a box titled Search Query. This is where you'll enter your queries. By default, this box contains a star symbol (*). This is the Lucene Query that shows all logs. The examples below will show you how to narrow this down, so you're only seeing the logs you want to see.
Generally, with Lucene, you'll be searching for a particular pattern of text and/or numbers in a specific field. For example, if you want to view all of the logs from a specific host, let's call it server.defense.com for example, you'd run the below Lucene query:
host:server.defense.com
Looking at the above, we've specified the field we'd like to search for, followed by a colon, followed by what we'd like to search for in this field. This will filter out any logs that don't match this pattern.
You don't have to specify a field when running a Lucene query. If you type in what you want to search for (e.g. server.defense.com or "Delete user"), all fields will be searched.
Querying data that includes spaces
The above syntax applies to search terms that don't contain any spaces. But what if the pattern you're looking for contains spaces? In this case, wrap your search term in quotation marks. For example, if we only want to view the logs where the field action, contains the string Delete user, use the below query:
action:"Delete user"
If it's the field, rather than the pattern that contains a space, this is handled differently. For example, if the field you're searching in is called host name, insert a backslash (\) just before the space as below:
host\ name:server.defense.com
Querying using wildcards
If you can see multiple fields that start with a common string, such as article.title, article.content and article.date, you can search within all of these fields at the same time by using a Wildcard. To do this, start your query with the text that each has in common, then add a backslash (\), followed by a star (*). For example, to search all of these fields for the string "test123", do:
article.\*:test123
Querying using the OR function
Another use case is if you want to search for multiple different strings at the same time. For example, if you want to search for all logs from several hosts, we encase the search terms in brackets and use the OR operator:
host:(server.defense.com OR server1.defense.com)
This will return any logs where the host field contains either server.defense.com or server1defense.com. You can do this for more than two strings, for example:
host:(server.defense.com OR server1.defense.com OR server2.defense.com OR server4.defense.com)
And that's it! You now understand the basics of querying your log data using Lucene Query Language 🎉
For further information and details on creating more complex queries, please see the Official Elasticsearch Documentation.