Kinro

The problem

Green dashboards. Broken orders.

Monitoring tells you when a process crashes or an endpoint returns 500. The bugs that cost real money do neither. Each service does its own part correctly, and the business process still fails in the gap between them. Customers notice first, support files a ticket, and engineers spend a day reading logs from five services to find one order.

no_duplicates

Charged twice

A retry captures the same payment again. Refunds, chargebacks, and a customer who stops trusting you.

billing payment.captured order=A1B2

billing payment.captured order=A1B2 ← again

chain

Paid, never shipped

The order is paid, but the warehouse event never arrives. Nobody sees it until the customer writes in.

orders order.created → billing payment.captured

shipping shipment.created missing after 30m

ordering

Shipped before payment

A race lets the parcel leave before the money clears. Goods go out on orders that later fail to pay.

shipping shipment.created 12:00:03

billing payment.captured 12:00:09

status_desync

Logs say paid, database says created

The service reported a status change that never reached the database. Reports, invoices and support all see a different truth.

log order.status_changed status=paid

db orders.status = 'created' after 60s

db_invariant

Data that breaks a rule

Paid orders with a zero amount, refunds above the original charge. Rows that should never exist, found on a schedule.

SELECT uid AS key FROM orders

WHERE status = 'paid' AND amount <= 0

your rule

Anything your business promises

Write the rule in Python: orders stuck in "created", a refund without a ticket, a flow across a dozen services.

class StuckInCreated(Rule):

ctx.violate('stuck for 30 minutes')

No rules needed

Useful on day one, before you write a rule

Most watchers start with an empty rules file. Kinro starts with your logs. It reads them as they are, learns what normal looks like, and only then proposes rules, which stay quiet until you accept them.

01 · Discover

See what is in your logs

Services, the ids that link them, flows with latencies, error bursts, retry loops, silent gaps and leaked secrets. One read-only pass, one masked report.

02 · Timeline

Follow one id

Type a job, request or entity id and get every event it touched, across services, in time order.

03 · Watch

Generic monitors

New error templates, error spikes, silent services and retry loops. Each service is learned for 24 hours, then you get one digest a day. Only the monitors you pick can page you.

04 · Suggest

Rules learned from your data

Chains, duplicates and ordering, with thresholds and evidence taken from your own flows. They start in shadow mode and page no one until you promote them.

Works with any logs

No log format to adopt first

Discover gives every service a tier. Higher tiers make the results more exact, but none is required. The report tells you what the next tier would add, service by service.

L0 · plain text

Any format

Templates are mined from the lines, and ids are read from the wording: k=v, /items/<id>, job <id>. Discover, timeline and the monitors all work here.

L1 · structured

JSON, logfmt or GELF with ids

Fields are read as they are, so keys and links between services no longer depend on how messages are worded.

L2 · Kinro contract

Recommended, optional

Explicit events and keys, numbered lines that reveal lost logs, and heartbeats that tell a quiet service from a stopped one. One formatter in the logging setup.

How it works

One timeline per id, across every service

Kinro sits next to your systems and only reads. It turns log lines (and, with custom rules, query rows) into observations, links them by key, and checks them as the data arrives: with generic monitors at first, then with the rules you accept or write.

01 · Read

Pull, never push

Logs come from Grafana Loki, Graylog or log files, in any format. With custom rules, data also comes from SELECT queries on a read-only Postgres or MySQL user. Your services keep running untouched.

02 · Correlate

Link by key

Discover finds the fields that share values across services and confirms them as keys. In a project file, a shared correlation id or a mapping such as uid in orders to external_id in billing does the same.

03 · Check

Judge on event time

Monitors and rules run on when things happened, not when logs arrived. A deadline fires only after every source has caught up, so slow logs do not raise false alarms.

04 · Alert

Send the evidence

Telegram or Slack gets the evidence: templates and masked examples for an insight, or the key, the timeline from every service and the exact step that broke for a rule.

orders LogContract · Loki

event order.created

cid 0192f0c4

keys uid=A1B2

data amount=10.50

billing legacy text · file

event payment.captured

cid —

keys external_id=a1b2

data card=[redacted]

shipping LogContract · Loki

event shipment.created

cid 0192f0c4

keys —

data carrier=dhl

uid ↔ external_id joins billing, both normalised to lower case cid joins shipping, even though it never logs the order id Result: one timeline for order A1B2

Kinro · #orders-alerts 12:30:09

[KINRO] order_flow v1
alert_id: 7f3c9e21a4d0b8e6
key: order=A1B2
missing step shipping/shipment.created within 0:30:00
expected: "shipment.created"
deadline: complete @ 2026-10-16T12:30:02Z
detected: 2026-10-16T12:30:09Z
timeline (2):
   2026-10-16T12:00:02Z orders order.created cid=0192f0c4… uid=A1B2 {"amount": "10.50"}
>> 2026-10-16T12:00:05Z billing payment.captured external_id=a1b2 {"amount": "10.50", "card": "[redacted]"}

The alert

Everything needed to act, in one message

R

Which promise broke

The rule name and version, so the owner knows what to check.

K

Which customer or order

The business key, ready to paste into your admin panel.

T

What happened, in order

The cross-service timeline with the broken step marked. No more grepping five log streams.

✓

Never lost

Alerts are saved before they are sent. After a crash one may arrive again, with the same alert id, so a repeat is easy to spot.

Get started

The first steps need no code

You need one Docker image, read access to your logs and four commands. Your services keep running as they are. Custom rules come later, if you want them.

Step 1

Run discover over your logs

Point kinro discover at a folder of log files, a Loki or a Graylog. It only reads, and it tells you what it found.

Services and their log tier, found without any configuration.
Keys that link services, confirmed by values they share.
Error bursts, retry loops, silent gaps, and leaked secrets or personal data.
Every value is masked in the report, the summary and the index.
It all ships in one image, kybalda/utyg:0.1.3; nothing else to install.

terminal

$ kinro discover --logs ./logs --out ./discovery
kinro discover: 3 services, 1214 records, 0.1 MB from 3 files in 0.1 s
services (tier, records, errors, entity ids):
  L1 core/indexer: 559 records, 84 errors; tenant
  L1 edge/api: 341 records, 0 errors; job_ref, tenant_id
  L0 notify/mailer: 314 records, 13 errors; job
keys across services: 2 (2 confirmed links, 1 unconfirmed candidates)
  k1 job: edge/api:job_ref, notify/mailer:job
  k2 tenant: core/indexer:tenant, edge/api:tenant_id
findings: 1 error bursts, 1 repeated failures, 4 top error templates, 1 silent gaps
  notify/mailer: 1 job values hit the same failure 5+ times (max 12) [2026-10-16T12:52:36Z .. 2026-10-16T13:03:36Z]
    job <hex> failed: upstream unavailable
secrets: 2, personal data: 2 (values masked)
  edge/api: credential `access_token` ****y0Ge in 2 records
wrote discovery/discovery.json, discovery/report.html and discovery/index.sqlite

Step 2

See what happened to one id

kinro timeline takes any id, such as a job, a request or an entity, and shows its events across every service in time order.

Ids are followed one hop through the confirmed cross-service keys.
Other ids in the same records are listed as related.
An email or a phone number is found through its keyed hash; it is never stored in clear.

terminal

$ kinro timeline 9d37ff01fe80 --index ./discovery --limit 4
kinro timeline 9d37ff01fe80: 14 events in 2 services (edge/api, notify/mailer), 2026-10-16T12:52:34.872Z .. 2026-10-16T13:04:36.488Z
showing the last 4 of 14 events (--limit)
key k1 job (confirmed link across services)
2026-10-16T13:01:36.488Z  notify/mailer  error    job 9d37ff01fe80 failed: upstream unavailable  [job]
2026-10-16T13:02:36.488Z  notify/mailer  error    job 9d37ff01fe80 failed: upstream unavailable  [job]
2026-10-16T13:03:36.488Z  notify/mailer  error    job 9d37ff01fe80 failed: upstream unavailable  [job]
2026-10-16T13:04:36.488Z  notify/mailer  info     job 9d37ff01fe80 delivered  [job]
related ids in the same records (keys followed one hop)
  b14aed54bb69e1f0  edge/api:trace  1 record

Step 3

Watch without rules

Start kinro run without a project file. It reads every service of the sources named in KINRO_* variables and runs four generic monitors on event time.

It learns each service for 24 hours before it reports anything.
One digest a day; only the monitors in KINRO_PAGE page at once.
Evidence is masked before it reaches the state or any chat.

terminal

$ export KINRO_LOKI_URL=https://loki.internal:3100 KINRO_PAGE=error_spike,silent_service
$ kinro check
zero-config: sources loki; monitors new_error_template, error_spike, silent_service, repeated_failure; learning 24h per service; paging error_spike, silent_service; digest 09:00 UTC; state kinro.db; 0 errors, 0 warnings
$ kinro run

the next morning, in Telegram

digest

[KINRO DIGEST] 2026-10-17 (09:00 UTC)
3 services watched (0 still learning), 1215 records, 97 errors
New insights: 3
  new_error_template (1):
    - core/indexer: new error template "connection refused" first seen 2026-10-16 10:30 UTC
  repeated_failure (1):
    - notify/mailer: job=****fe80 hit "job <hex> failed: upstream unavailable" 5 times in 4m (a retry loop?)
  silent_service (1, 1 paged):
    - edge/api: no records since 2026-10-16 13:58:58 UTC (58m); its longest learned pause is 7m

Step 4

Accept the suggested rules

kinro suggest turns the discovery into a project file. Its rules come with thresholds learned from your flows and their evidence as comments, and the file is validated before it is written.

chain, no_duplicates and ordering, with a TemplateParser per service.
Every rule has shadow=True, so its alerts go to a file and the log only.
To promote a rule, delete shadow=True once its alerts look right.

terminal

$ kinro suggest --from ./discovery --out project.py --logs-root /logs
suggest: 2 services (1 left out), 1 keys (job), 4 rules (1 chain, 2 no_duplicates, 1 ordering), 3 candidates skipped (support: 3); 300 entities, 0 ambiguous
wrote project.py: 2 services, 4 shadow rules; validated. Next: kinro check --project project.py, then kinro replay --project project.py
$ kinro replay --project project.py
SHADOW chain_job_submitted_2c772f job=9d37ff01fe80 008fe6efee80e596: missing step notify/mailer/job_delivered within 0:00:11
replay: 1 violations (1 shadow), 0 meta-alerts, 0 late follow-ups; 600 observations, 0 late, 0 rule errors

a rule in project.py

project.py

    # 'job_submitted' (edge/api) is followed by 'job_delivered' (notify/mailer) within 11s: 298 of 299 flows by job
    # (cross-service key, confidence 0.997); latency to the slowest step p50 1.5s, p95 1.65s, p99 1.75s
    chain(
        'chain_job_submitted_2c772f',
        key='job',
        steps=[API_JOB_SUBMITTED, MAILER_JOB_DELIVERED],
        within=timedelta(seconds=11),
        shadow=True,
    ),

Custom rules · optional

Your own rules, when you need them

Some checks only you know: a database row that must match a log line, a payload field, a promise between teams. Write them in Python next to the suggested ones, with the same SDK, in one project file.

Custom rules · 1

Emit the log contract, if you can

Optional and recommended: the contract is the top log tier. Set ContractFormatter on the Loki handler and start a heartbeat. Then name the business event in the log calls you already make.

One JSON line per event: time, service, event, correlation id, keys, data.
Heartbeats tell Kinro that a quiet service is alive, not lost.
Per-process sequence numbers reveal lines that never reached Loki.

orders/core/logging.py

from kinro.contract import ContractFormatter, start_heartbeat

loki = logging_loki.LokiHandler(url=LOKI_URL, tags={'service': 'orders'}, version='1')
loki.setFormatter(ContractFormatter('orders'))
logging.getLogger().addHandler(loki)
start_heartbeat(logging.getLogger())

logger.info('order %s created', uid, extra={
    'event': 'order.created',
    'cid': cid,                     # propagated to billing and shipping
    'keys': {'uid': uid},
    'data': {'amount': str(amount), 'email': email},
})

lands in Loki as

loki · {service="orders"}

{"kinro_v": 1, "ts": "2026-10-16T12:00:02.114+00:00", "svc": "orders",
 "event": "order.created", "cid": "0192f0c4-7c1e-7d3a-9b1e-5f2a1c3d4e5f",
 "keys": {"uid": "A1B2"}, "data": {"amount": "10.50", "email": "ann@example.com"},
 "inst": "orders-7f9c:12", "seq": 41}

Custom rules · 2

Keep legacy formats with a custom parser

A service you cannot change still counts. Describe its log line with a regular expression, map its messages to events, and Kinro reads the file as it grows.

RegexParser for any text format, with local time zones.
Or write your own parser class for anything else.
redact=[...] hides card numbers and emails before anything is stored or sent.
Rotation by rename or copytruncate, with no lines lost.

project.py

project.service(
    'billing',
    LogSource(
        '/logs/billing/billing.log',
        parser=RegexParser(
            r'^(?P<ts>\S+ \S+) (?P<level>\w+) \[billing\] (?P<event>payment captured)'
            r' ext=(?P<external_id>\w+) amount=(?P<amount>[\d.]+) card=(?P<card>\S+)$',
            events={'payment captured': 'payment.captured'},
            ts_format='%Y-%m-%d %H:%M:%S,%f',
            tz='Europe/Kyiv',
        ),
    ),
    keys={'order': Field('external_id', normalize=str.lower)},
    redact=['card'],
)

billing.log gets a new line

tail -f /logs/billing/billing.log

2026-10-16 15:00:05,532 INFO [billing] payment captured ext=A1B2 amount=10.50 card=4111111111111111

Kinro reads it as

observation

{"service": "billing", "event": "payment.captured",
 "ts": "2026-10-16T12:00:05.532+00:00",
 "keys": {"order": "a1b2"},
 "payload": {"amount": "10.50", "card": "[redacted]"}}

Custom rules · 3

Describe the flow in one file

List your services, say where their logs live and how they relate, then add rules. Built-in rules cover the common failures in one line each.

Loki sources take a selector, a read-only token and a query budget.
Databases are reached through a role that can only SELECT.
The same file runs in production, in replay and in tests.

project.py

def loki(service: str) -> LokiLogs:
    return LokiLogs(LokiConfig(selector=f'{{service="{service}"}}'), url_env='LOKI_URL', token_env='LOKI_TOKEN')


project = Project('production', state_path='/state/kinro.db')
project.service('orders', loki('orders'), keys={'order': Field('uid', normalize=str.lower)}, redact=['email'])
project.service('shipping', loki('shipping'))
project.db('orders_db', url_env='ORDERS_DB_URL', pgbouncer=True)

CREATED = on('orders', 'order.created')
PAID = on('billing', 'payment.captured')
SHIPPED = on('shipping', 'shipment.created')

project.add(
    chain('order_flow', key='order', steps=[CREATED, PAID, SHIPPED], within=timedelta(minutes=30)),
    no_duplicates('single_payment', key='order', trigger=PAID, within=timedelta(hours=1)),
    ordering('paid_before_shipped', key='order', first=PAID, then=SHIPPED),
    status_desync('order_status_in_db', key='order', trigger=on('orders', 'order.status_changed'),
                  field='status', db='orders_db', settle=timedelta(seconds=60),
                  lookup='SELECT uid AS key, status, updated_at FROM orders WHERE uid = ANY(:keys)'),
)

Custom rules · 4

Check, replay, run

kinro check validates rules, sources and settings without connecting anywhere. kinro replay runs your rules over last hour's logs and sends nothing. Then start it for real.

One process and one SQLite file. No brokers, no cluster.
Ships as a container: mount logs read-only, pass the env.
A stats line every minute shows lag and anything it could not verify.

terminal

$ kinro check --project project.py
production: 7 rules, 3 services, 1 dbs; 0 errors, 0 warnings
$ kinro replay --project project.py --since 3600
order_flow order=a1b2 7f3c9e21a4d0b8e6: missing step shipping/shipment.created within 0:30:00
replay: 1 violations, 0 meta-alerts, 0 late follow-ups; 41812 observations, 0 late, 0 rule errors
replay: DB schedules and lookups skipped (orders_db); pass --db to run them
$ kinro run --project project.py
2026-10-16 12:00:00,412 | INFO     | kinro.engine.runtime | kinro running: project production, 4 sources, channels telegram, slack, file, log, shadow
2026-10-16 12:01:00,415 | INFO     | kinro.engine.runtime | runtime stats: {"queue": 0, "queue_max": 1000, "queue_high_water": 12, "batches": 240, "observations": 1830, "late_observations": 0, "rule_errors": 0, "violations": 0, "desync_unverifiable": 0, "chain_unverifiable": 0, "link_conflicts": 0, "pending_keys": 610, "db_queries": 14, "rotation_gaps": 0, "parse_errors": 0, "lag_seconds": {"billing": 0.4, "orders": 0.8, "orders_db": 12.1, "shipping": 0.9}}

Custom rules · 5

Your business rules, in plain Python

The built-in rules are written with the same SDK you get. A rule reacts to observations, query rows and its own deadlines, keeps state per key, and raises a violation with the step that broke.

chainno_duplicatesorderingstatus_desyncdb_invariantRule

Test rules without Loki or a database: RuleHarness feeds observations, moves the clock and returns the violations. Rule authors install only kinro-sdk==0.1.3, matching the image.

rules/stuck.py

class StuckInCreated(Rule):
    key = 'order'
    version = 1
    schedules = (Query(
        'orders_db', name='stuck_created', every=timedelta(minutes=5),
        sql="SELECT uid AS key, created_at FROM orders WHERE status = 'created' "
            "AND created_at < now() - interval '30 minutes' LIMIT 100",
    ),)

    def on_observation(self, ctx: RuleContext, obs: Observation) -> None:
        if obs.event == self.schedules[0].row_event:
            ctx.violate('order stuck in created for more than 30 minutes', broken_step=obs)
            ctx.close()

Safe by design

It watches production. It never touches it.

Read-only log stores

Kinro sends Loki only GET requests (query_range; discover also reads labels, volumes and patterns) and Graylog only its search API, under a request budget set with max_qps. Any other request fails before it is sent.

Read-only database

A Postgres role or MySQL user with SELECT grants and a read-only session. Any SQL that is not a single SELECT is rejected before start.

Read-only files

Log directories are mounted :ro. Kinro keeps its own cursor and never moves or truncates your files.

No personal data in chats

Discover, timeline and the monitors mask secrets, emails, phones and names on their own, showing at most the last four characters. With rules, fields listed in redact are masked as soon as a line is read.

Survives restarts

Cursors, rule state and pending alerts commit together. After a crash it resumes where it stopped and loses nothing; an alert resent after a crash keeps its alert id.

Honest when blind

If logs go missing, you get one gap alert instead of a flood of false "missing step" reports.

70/70injected bugs caught, each by exactly one rule

0false alerts across clean runs, rotations and a hard kill

<8sfrom a missed deadline to the message in Telegram (p95)

20MBof log files per minute in the throughput test, under a second of lag

Bugs, false alerts and latency: measured on the Kinro demo stand, three services with Grafana Loki and PostgreSQL in Docker, bugs injected on purpose, with hand-written rules. Throughput: a 5-minute test tailing local log files. Without writing a rule: rules suggested from clean demo traffic caught the 40 of those bugs that show in the logs, with no alert on clean traffic (the other 30 need a database or payload check); and the generic monitors, with every monitor paging, raised 1 insight over 14 simulated days of healthy synthetic services, for a rare error planted on day 3.

Rollout

Start with a shadow week

Run Kinro beside production for a week before anyone gets paged. Monitors report to a daily digest and suggested rules start in shadow mode. Tune what is noisy, then switch on paging.

Day 0

Deploy quietly

Start with kinro run and no paging. Shadow rules write to a file and the log only, never to Telegram or Slack.

Days 1–7

Review and tune

Mark each alert as a real bug or noise. Adjust windows and check every change with kinro replay. Target: at most 3 false alerts a day.

Day 8

Go live

Add monitors to KINRO_PAGE, delete shadow=True from the rules you trust, and restart. The alert file stays as an audit trail.

Find the bugs before your customers do

Point Kinro at the logs you already have, read what it finds, let it watch without rules, and accept the rules it learns. Write your own when you need them.

Get started Read the quickstart

Catch the bugs that hide between your services

Green dashboards. Broken orders.

Charged twice

Paid, never shipped

Shipped before payment

Logs say paid, database says created

Data that breaks a rule

Anything your business promises

Useful on day one, before you write a rule

See what is in your logs

Follow one id

Generic monitors

Rules learned from your data

No log format to adopt first

Any format

JSON, logfmt or GELF with ids

Recommended, optional

One timeline per id, across every service

Pull, never push

Link by key

Judge on event time

Send the evidence

Everything needed to act, in one message

Which promise broke

Which customer or order

What happened, in order

Never lost

The first steps need no code

Run discover over your logs

See what happened to one id

Watch without rules

Accept the suggested rules

Your own rules, when you need them

Emit the log contract, if you can

Keep legacy formats with a custom parser

Describe the flow in one file

Check, replay, run

Your business rules, in plain Python

It watches production. It never touches it.

Read-only log stores

Read-only database

Read-only files

No personal data in chats

Survives restarts

Honest when blind

Start with a shadow week

Deploy quietly

Review and tune

Go live

Find the bugs before your customers do