Data Pipeline initiatives allow companies to generate and act on insights from their investment in Big Data initiatives. By radically transforming the velocity at which applications and analytics can integrate new data, organizations can more easily react to new opportunities, make decisions, and transform key business processes.

BigID pioneered data discovery for personal data at rest – and we’re now extending those discovery and data intelligence capabilities to data in motion. The BigID platform now supports data pipelines so that companies can monitor sensitive data in motion on data streaming platforms including Kafka, Kafka Connect, AWS Kinesis, and FTP. Discover and classify data in motion, integrate consumer consent correlation across data streaming platforms, and extend privacy insight to high-speed pipelines.

The Challenge of Data Pipelines

Data streaming capabilities from tools like Apache Kafka, AWS Kinesis, and Confluent’s Kafka Connect enable organizations to utilize the data from existing mobile applications, IoT devices, business applications, data lakes, and other event sources to rapidly gain insight and trigger responsive business processes.

Global privacy regulations such as CCPA and GDPR, meanwhile, have introduced specific requirements for protecting new categories of personal information – while limiting how it can be used, shared or processed. To meet these regulations and approach data processing ethically, it’s critical that organizations integrate privacy insights into data pipeline initiatives.

Fines may have a material impact, but customers and consumers will move elsewhere if they lose trust in a company’s brand because of privacy violations.

Given the velocity, volume, and complexity of data flowing through today’s data pipelines, it’s a huge challenge for companies to ensure that they are doing the right thing with data – both in respect to data privacy, and by operating with privacy by design principles when developers and analytics teams build applications. In order to do so, organizations need to be able to:

– Identify sensitive personal data in motion (and at rest)
– Understand how it originated,
– Correlate it back to a set of identities for personal information classification, and
– Govern usage against consent purposes are among the challenges.

The stakes are high.

For one thing, losing track of personal data as it streams across the enterprise – or worse, using it in ways that violate consumer privacy protection – can undermine trust in ways that far exceed the cost of regulatory fines and statutory civil liabilities.

Enterprises can also lose their competitive advantage because the risk of either falling foul of regulations or utilizing personal information in ways that were not intended is too high to leverage data streams.

In an instant, almost as quickly as data can stream, trust can be irreparably broken between a company and its customers – even if the purpose was not nefarious.

You Don’t Need to Choose Between Privacy and Innovation

To help companies balance the drive for data pipeline-driven innovation with the need to protect sensitive personal information, BigID has introduced the industry’s first privacy-aware Data Pipeline Discovery solution.

The BigID Platform allows organizations to monitor sensitive PI and PII data transfers at scale, and govern consumer consent across high-speed data pipelines to help organizations comply with global privacy regulations. BigID’s insights can be scoped down to specific data stream ‘conversations’ so that policies can be monitored on a granular basis. The discovery is performed via APIs, and can be deployed at data lake ingress and egress points.

BigID’s unique data in motion capabilities now include:

– The ability to scan data in motion for data streaming solutions including Kafka, Kafka Connect, AWS Kinesis and FTP for direct visibility, data insight, and data intelligence.

– The industry’s broadest support for data sources residing in the data center or the cloud, and data in motion as well as data at rest. Support covers unstructured files, structured databases, data lakes, big data, data warehouses, mainframes, applications like SAP, SaaS applications such as Salesforce, and cloud environments including AWS, Azure and others.

– Near real-time population of personal data inventory as data is streamed into the data pipeline, providing fast time-to-value by eliminating the need to rely on a full scan of the data at rest.

– The ability to discover not only personally identifiable information (PII), but contextual personal information (PI), as defined by the new wave of privacy regulations.

– Classification, cataloging and critical correlation for tying data attributes to a person — essential steps for satisfying personal data rights as defined by CCPA and GDPR.

– The ability to correlate consent agreement logs and preferences to individuals and their data for governance.

– Monitoring and reporting for third party data transfers and use of exogenous data for applications and analytics

Privacy Made Practical

BigID’s new capabilities make privacy by design principles a practical reality for data pipeline initiatives – and enable better reporting and monitoring for compliance. The privacy office can work more productively with engineers, data scientists, and application developers to design and deploy new applications and services that capitalize on big data with agility, all with the confidence that sensitive information will be protected.

Privacy by design can be transformed from a set of abstract principles to a well structured and well understood set of privacy by engineering processes. Rather than rely on the privacy office to flag potential misuse of personal information in pipeline initiatives after the fact (or once analytics programs are already deployed), enterprises can bake privacy insights into the lifecycle.

BigID’s discovery and classification insights can be used to document that data is being used according to policy – and empower privacy, data science and engineering teams to take immediate action if it is isn’t. In this critical way, BigID helps organizations take an important step in breaking down silos and unifying the enterprise to ensure privacy and preserve trust.

Get a custom demo to more about BigID’s data pipeline discovery capabilities.