Successfully reported this slideshow.
We use your Verifieduser profile and activity data to personalize ads and to show you more relevant ads.
Pinot
Kishore Gopalakrishna
Tuesday, August 18, 15
Agenda
• Pinot @ Verifieduser - Current
• Pinot - Architecture
• Pinot Operations
• Pinot @ Verifieduser - Future
Tuesday, August ...
WVMP
Tuesday, August 18, 15
Slice and Dice Metrics
Tuesday, August 18, 15
Pinot @ Verifieduser
Customers Members Internal tools
Tuesday, August 18, 15
• 100B documents
• 1B documents ingested per day
• 100M queries per day
• 10’s of ms latency
• 30 tables in prod, 250 * 3 ...
Key features
SQL-like
interface
Columnar
storage and
indexing
Real-time
data load
Tuesday, August 18, 15
(S)QL: Filters and Aggs
SELECT count(*)
FROM companyFollowHistoricalEvents
WHERE entityId = 121011 AND
'day' >= 15949 AND ...
(S)QL: Group By
SELECT count(*)
FROM companyFollowHistoricalEvents
WHERE entityId = 121011 AND
'day' >= 15949 AND 'day' <=...
(S)QL: ORDER BY and LIMIT
SELECT *
FROM companyFollowHistoricalEvents
WHERE entityId = 121011 AND
entityId = 1000 AND
acti...
Whats not supported
• JOIN: unpredictable performance
• NOT A SOURCE OF TRUTH
• Mutation
Tuesday, August 18, 15
Pinot
• Data flow
• Query Execution
• How to use/operate
• Pinot @ Verifieduser - Future
Tuesday, August 18, 15
Broker Helix
Real
time
Historical
Kafka Hadoop
Pinot
Architecture
Queries
Raw
Data
Tuesday, August 18, 15
Pinot
• Pinot segments
Tuesday, August 18, 15
Pinot Segment layout: Columnar storage
Tuesday, August 18, 15
Pinot Segment layout: Sorted Forward Index
Tuesday, August 18, 15
Pinot Segment layout: Other techniques
• Indexes: Inverted index, Bitmap, RoaringBitmap
• Compression: Dictionary Encoding...
Data aware
pre-computation
Star tree Index
Tuesday, August 18, 15
Pinot
• Query Execution
Tuesday, August 18, 15
Pinot Query Execution: Distributed
Servers
S1
S3
S2
S1
S3
S2
Helix
Brokers
Tuesday, August 18, 15
Pinot Query Execution: Distributed
Servers
1.Query
S1
S3
S2
S1
S3
S2
Helix
Brokers
Tuesday, August 18, 15
Pinot Query Execution: Distributed
Servers
1.Query
S1
S3
S2
S1
S3
S2
Helix
2. Fetch routing table from HelixBrokers
Tuesda...
Pinot Query Execution: Distributed
Servers
1.Query
S1
S3
S2
S1
S3
S2
Helix
2. Fetch routing table from HelixBrokers
3. Sca...
Pinot Query Execution: Distributed
Servers
1.Query
S1
S3
S2
S1
S3
S2
Helix
2. Fetch routing table from HelixBrokers
3. Sca...
Pinot Query Execution: Distributed
Servers
1.Query
S1
S3
S2
S1
S3
S2
Helix
2. Fetch routing table from HelixBrokers
3. Sca...
Pinot Query Execution: Distributed
Servers
1.Query
S1
S3
S2
S1
S3
S2
Helix
2. Fetch routing table from HelixBrokers
3. Sca...
Pinot Query Execution: Single Node Architecture
EXECUTION ENGINE
INVERTED
INDEX
BITMAP
INDEX
COLUMN FORMAT
PLANNER
Tuesday...
Pinot Query Execution: Single Node Architecture
SELECT
campaignId,
sum(clicks)
FROM Table A
WHERE
accountId = 121011
AND
'...
Pinot
• Operations
Tuesday, August 18, 15
Cluster Management: Deployment
Helix
Brokers
Servers
• Brokers and Servers register themselves in Helix
• All servers star...
On boarding new use case
Helix
Brokers
Servers
XLNT XLNT
XLNT
Create Table
command
Controller
XLNT
XLNTTag
Servers
TableNa...
Segment Assignment
Servers
S3
S2
S1
Upload Segment S2
S1
S3
S2
S1
S3
Helix
Brokers
Copies
TableName
2
XLNT_T1
Controller
T...
• AUTO recovery mode: Automatically redistribute
segments on failure/addition of new nodes
• Custom mode: Run in degraded ...
Pinot vs Druid
Druid Pinot
Architecture
Realtime + Offline,
Realtime only
Realtime + Offline
Realtime only -> consistency is...
• Documentation & tooling
• In progress - consistency among real time replicas.
• Improve cost to serve - leverage SSD, pa...
Thank You
30
Tuesday, August 18, 15
Upcoming info
Loading in …5
×

Pinot Query Execution: Distributed Servers Pinot: Realtime Distributed OLAP datastore

695,029 views

Published on

Pinot Query Execution: Distributed
Servers
1.Query
S1
S3
S2
S1
S3
S2
Helix
Brokers
Tuesday, August 18, 15

Published in: Technology
  • Theo phong tục và quan niệm truyền thống dân gian Việt Nam khi trẻ vừa tròn 1 tuổi thì gia đình sẽ tổ chức một buổi lễ cúng thôi nôi. Việc làm thôi nôi cho bé mang ý nghĩa quan trọng, đánh dấu bước đầu phát triển của trẻ nhỏ. Thông qua buổi lễ cúng này, những điều tốt đẹp nhất sẽ được khấn nguyện đến các vị Tôn Thần, Tiên Nương,…. và trẻ sẽ lớn lên bình an, mạnh khỏe. Xem xôi chè một chữ tâm tại đây.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • my buddy's friend makes $96 hourly on the internet. She has been with out artwork for five months however final month her charge emerge as $12747 really on foot on the internet for some hours. study greater on this net internet site..... HERE ☛  www.joblinks99.com
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Dù căn cứ theo đặc tính nào để có thể đưa ra sự chọn lọc chính xác nhất về các loại yến thì yếu tố chất lượng vẫn phải đặt lên hàng đầu nhằm đảm bảo nguồn dưỡng chất quý giá không bị hao hụt hay mất đi trong quá trình chế biến và sử dụng cho các nhóm đối tượng khác nhau. Hãy cùng chúng tôi khám phái bài viết phân loại yến sào tại Yến sào bảo vy.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Để có thể nhận biết và phân biệt các loại yến sào một cách chính xác nhất, chúng ta có thể căn cứ vào nhiều yếu tố khác nhau. Theo nguồn gốc, Theo màu sắc, Theo quan niệm . Tham khảo tại Yến sào bảo vy
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • like brendA responded i'm shocked thAt A stAy At home mom Able to eArn $7047 in one month on the computer. hAve you reAd this web site    www.hit2day.com
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

×