Module com.darkcollective.relix.connectors.std
Package com.darkcollective.relix.connectors.std.internal

Class LogConnector

java.lang.Object
com.darkcollective.relix.connectors.std.internal.LogConnector
All Implemented Interfaces:
RelixConnector, AutoCloseable
public final class LogConnector extends Object implements RelixConnector
Reads a web server's access log as a relation: connection X from log { … }. 

 connection logs from log { path: "/var/log/nginx", format: "combined" };
 source Access from logs { table: "access.log",
     schema: { host: STRING, at: TIMESTAMP, method: STRING, path: STRING,
               status: NUMBER, bytes: NUMBER, referer: STRING, agent: STRING } };

The format decides the fields, and their types

format is common, combined, or the server's own format string, pasted from its configuration: Apache's LogFormat or nginx's log_format; it defaults to common, and clf is another name for the log type. What a log is worth to a query is the typing: the request time is a TIMESTAMP, the status and byte count are numbers, the request line is split into method, path and protocol, and a - is NULL wherever it appears.

The declared schema decides the heading

Every column is filled by name, and a column the format does not produce is NULL, the rule the GEDCOM connector follows. A dotted reference, with nothing declared, gets every field the format produces.

A directory holds several logs

When the connection's file is a directory, table names a file in it, so access.log and a rotated access.log.2.gz are two tables of one connection. When it is a file, table names that file. A gzip file is read transparently.

Reading

A log is streamed a line at a time rather than read whole, so the relation holds an open file until the query closes it. A line the format does not match is an error naming its line number, unless the connection says onError: "skip": then it is left out, and the number skipped is logged when the file is closed. Blank lines are not records and are passed over.

  • Constructor Details

    • LogConnector

      public LogConnector()
      Public no-arg constructor required for ServiceLoader discovery.

  • Method Details

    • handles

      public Set<String> handles()
      Description copied from interface: RelixConnector
      The type tokens this connector handles, lower-cased — e.g. {"jdbc"}, {"csv"}, {"mongodb"}. The registry uses these for dispatch.
      Specified by:
      handles in interface RelixConnector
      Returns:
      a non-empty set of handled type tokens

    • tableSchema

      public Optional<Schema> tableSchema(ConnectorConfig config, String table)
      Introspects the schema of a table, when the backend supports it. The default returns empty — "not available", in which case the declared schema is used as-is.

      Every field the format produces, at its natural type. Described from the format alone, so an unreadable file does not stop analysis.

      Specified by:
      tableSchema in interface RelixConnector
      Parameters:
      config - the connector configuration
      table - the table/collection name
      Returns:
      the introspected schema, or empty when unavailable

    • open

      public Stream<Row> open(ConnectorConfig config, String table, Schema schema)
      Description copied from interface: RelixConnector
      Opens a stream of rows for a named table/collection/endpoint. The caller closes the returned stream.
      Specified by:
      open in interface RelixConnector
      Parameters:
      config - the connector configuration for this relation
      table - the table/collection/endpoint name within the source
      schema - the expected output schema (column order and types)
      Returns:
      a stream of rows; the caller is responsible for closing it