Know Your Data in Kafka
Discover sensitive and personal data in Kafka
Know Your Data in Kafka
Map and monitor Kafka streams for sensitive data with BigID.
Leveraging an agentless connector, BigID scans a sample quantity of messages on every poll interval so you can quickly discover sensitive and personal information using minimal resources and storage.
BigID supports both Core Kafka and Confluent Kafka and can use its schema management to work with Avro message serialization. BigID can add additional scanners and correlators on the same queue when the amount of data is too big for one scanner and correlator.
Advantages of Kafka
Low Latency, High Throughput
Optimized to process a high volume and velocity of data messages—thousands per second—with little delay.
Durability
Offers the replication feature, allowing data or messages to persist longer on the cluster over a disk.
Scalability
Handles a large number of messages simultaneously, making it a scalable software product.
Real-time Handling
Supports real-time data pipelines, including processors, analytics, storage.
Open Source
Available to businesses for free, and open to public collaboration.
Distributed System
Contains a distributed architecture, allowing for partitioning and replication.
About Kafka
Apache Kafka is an open-source, distributed, stream-processing software platform developed by LinkedIn and donated to the Apache Software Foundation.
A streaming platform has three key capabilities:
- Publish and subscribe to streams of records, similar to a message queue
- Store streams of records in a fault-tolerant, durable way
- Process streams of records as they occur
Kafka is used by many companies, including Airbnb, Uber, and Netflix, to replicate data between nodes, re-sync for nodes, and restore state. While Kafka is mostly used for real-time data analytics and stream processing, it can also be used for log aggregation, messaging, click-stream tracking, and audit trails.
