Successfully reported this slideshow.
We use your Verifieduser profile and activity data to personalize ads and to show you more relevant ads.
Pinot
Kishore Gopalakrishna
Tuesday, August 18, 15
Agenda
• Pinot @ Verifieduser - Current
• Pinot - Architecture
• Pinot Operations
• Pinot @ Verifieduser - Future
Tuesday, August ...
WVMP
Tuesday, August 18, 15
Slice and Dice Metrics
Tuesday, August 18, 15
Pinot @ Verifieduser
Customers Members Internal tools
Tuesday, August 18, 15
• 100B documents
• 1B documents ingested per day
• 100M queries per day
• 10’s of ms latency
• 30 tables in prod, 250 * 3 ...
Key features
SQL-like
interface
Columnar
storage and
indexing
Real-time
data load
Tuesday, August 18, 15
(S)QL: Filters and Aggs
SELECT count(*)
FROM companyFollowHistoricalEvents
WHERE entityId = 121011 AND
'day' >= 15949 AND ...
(S)QL: Group By
SELECT count(*)
FROM companyFollowHistoricalEvents
WHERE entityId = 121011 AND
'day' >= 15949 AND 'day' <=...
(S)QL: ORDER BY and LIMIT
SELECT *
FROM companyFollowHistoricalEvents
WHERE entityId = 121011 AND
entityId = 1000 AND
acti...
Whats not supported
• JOIN: unpredictable performance
• NOT A SOURCE OF TRUTH
• Mutation
Tuesday, August 18, 15
Pinot
• Data flow
• Query Execution
• How to use/operate
• Pinot @ Verifieduser - Future
Tuesday, August 18, 15
Broker Helix
Real
time
Historical
Kafka Hadoop
Pinot
Architecture
Queries
Raw
Data
Tuesday, August 18, 15
Pinot
• Pinot segments
Tuesday, August 18, 15
Pinot Segment layout: Columnar storage
Tuesday, August 18, 15
Pinot Segment layout: Sorted Forward Index
Tuesday, August 18, 15
Pinot Segment layout: Other techniques
• Indexes: Inverted index, Bitmap, RoaringBitmap
• Compression: Dictionary Encoding...
Data aware
pre-computation
Star tree Index
Tuesday, August 18, 15
Pinot
• Query Execution
Tuesday, August 18, 15
Pinot Query Execution: Distributed
Servers
S1
S3
S2
S1
S3
S2
Helix
Brokers
Tuesday, August 18, 15
Pinot Query Execution: Distributed
Servers
1.Query
S1
S3
S2
S1
S3
S2
Helix
Brokers
Tuesday, August 18, 15
Pinot Query Execution: Distributed
Servers
1.Query
S1
S3
S2
S1
S3
S2
Helix
2. Fetch routing table from HelixBrokers
Tuesda...
Pinot Query Execution: Distributed
Servers
1.Query
S1
S3
S2
S1
S3
S2
Helix
2. Fetch routing table from HelixBrokers
3. Sca...
Pinot Query Execution: Distributed
Servers
1.Query
S1
S3
S2
S1
S3
S2
Helix
2. Fetch routing table from HelixBrokers
3. Sca...
Pinot Query Execution: Distributed
Servers
1.Query
S1
S3
S2
S1
S3
S2
Helix
2. Fetch routing table from HelixBrokers
3. Sca...
Pinot Query Execution: Distributed
Servers
1.Query
S1
S3
S2
S1
S3
S2
Helix
2. Fetch routing table from HelixBrokers
3. Sca...
Pinot Query Execution: Single Node Architecture
EXECUTION ENGINE
INVERTED
INDEX
BITMAP
INDEX
COLUMN FORMAT
PLANNER
Tuesday...
Pinot Query Execution: Single Node Architecture
SELECT
campaignId,
sum(clicks)
FROM Table A
WHERE
accountId = 121011
AND
'...
Pinot
• Operations
Tuesday, August 18, 15
Cluster Management: Deployment
Helix
Brokers
Servers
• Brokers and Servers register themselves in Helix
• All servers star...
On boarding new use case
Helix
Brokers
Servers
XLNT XLNT
XLNT
Create Table
command
Controller
XLNT
XLNTTag
Servers
TableNa...
Segment Assignment
Servers
S3
S2
S1
Upload Segment S2
S1
S3
S2
S1
S3
Helix
Brokers
Copies
TableName
2
XLNT_T1
Controller
T...
• AUTO recovery mode: Automatically redistribute
segments on failure/addition of new nodes
• Custom mode: Run in degraded ...
Pinot vs Druid
Druid Pinot
Architecture
Realtime + Offline,
Realtime only
Realtime + Offline
Realtime only -> consistency is...
• Documentation & tooling
• In progress - consistency among real time replicas.
• Improve cost to serve - leverage SSD, pa...
Thank You
30
Tuesday, August 18, 15
Upcoming info
Loading in …5
×

Pinot • Pinot segments Tuesday, Pinot: Realtime Distributed OLAP datastore

707,962 views

Published on

Pinot
• Pinot segments
Tuesday, August 18, 15

Published in: Technology
  • Start working at home with Google! Just work for few hours and have more time with friends and family. I earn up to $4500 per week. It's a great work at home opportunity. I can't believe how easy it was once I tried it out. Visit for more details.☛☛☛☛☛☛☛☛☛☛☛☛☛☛☛☛☛☛☛☛☛☛☛☛ www.Buzz90.com
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • My life Was devastated! But All Thanks To Dr happy Who Brought Back My girlfriend after cheating on me and ask for a breakup, Alexis scorpio from FL Usa. I don't believe at first i try to get back with her but all she told me was she’s with someone else. that she is no longer interested in marrying me at that point i was heart broken coz i love my girlfriend so much that i could not let go off her all of a sudden she left me, i really love her and never can imagine my life without her. not until i came across a powerful real spell caster Dr happy who promise me 24hours urgent love spell to get back with my girlfriend, good twenty-four {24} hours. hmm-mm, it was a good night time at 11:pm within the days that Dr happy told me that my girlfriend will be back, at first i heard the bell rings getting close to my door i heard someone saying honey!!!, it sound familiar i opened the door and i saw my girlfriend standing and weeping in front of me. i was not surprised because its all i have been praying for her to come back home. Guess what 2 weeks after she noticed her system and her body temperature was changed and i took her to clinic for check up and the doctor told me that there is life in her which means she was pregnant i really wants to use this opportunity to thanks Dr happy so much and the love page that directed me to Dr happy if you have any problem getting your ex back, or predicament that is worse or exactly like this you have been into, Dr happy on [email protected] Whatsapp/cal +2348133873774 visit his website...happylovespell2.webnode.com/ about his love spell Blogs site.http://happylovespel.blogspot.com.ng/
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • My life Was devastated! But All Thanks To Dr happy Who Brought Back My girlfriend after cheating on me and ask for a breakup, Alexis scorpio from FL Usa. I don't believe at first i try to get back with her but all she told me was she’s with someone else. that she is no longer interested in marrying me at that point i was heart broken coz i love my girlfriend so much that i could not let go off her all of a sudden she left me, i really love her and never can imagine my life without her. not until i came across a powerful real spell caster Dr happy who promise me 24hours urgent love spell to get back with my girlfriend, good twenty-four {24} hours. hmm-mm, it was a good night time at 11:pm within the days that Dr happy told me that my girlfriend will be back, at first i heard the bell rings getting close to my door i heard someone saying honey!!!, it sound familiar i opened the door and i saw my girlfriend standing and weeping in front of me. i was not surprised because its all i have been praying for her to come back home. Guess what 2 weeks after she noticed her system and her body temperature was changed and i took her to clinic for check up and the doctor told me that there is life in her which means she was pregnant i really wants to use this opportunity to thanks Dr happy so much and the love page that directed me to Dr happy if you have any problem getting your ex back, or predicament that is worse or exactly like this you have been into, Dr happy on [email protected] Whatsapp/cal +2348133873774 visit his website...happylovespell2.webnode.com/ about his love spell Blogs site.http://happylovespel.blogspot.com.ng/
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Start working at home with Google! Just work for few hours and have more time with friends and family. I earn up to $4500 per week. It's a great work at home opportunity. I can't believe how easy it was once I tried it out. Visit for more details.☛☛☛☛☛☛☛☛☛☛☛☛☛☛☛☛☛☛☛☛☛☛☛☛ www.Buzz90.com
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • My ­n­e­i­g­h­b­or's ­m­ot­h­er ­m­A­k­es $64 ­h­our­ly ­o­n t­h­e ­l­A­pt­o­p. S­h­e has ­b­e­e­n ­out ­o­f w­or­k ­f­or ­f­iv­e ­m­o­nt­hs ­but ­l­Ast ­m­o­nt­h ­­p­Ay­m­e­nt w­As $15080 just w­or­k­i­n­g ­o­n t­h­e l­A­pt­o­p ­f­or ­A ­f­ew ­h­ours. ­g­o t­o t­h­is w­e­b s­it­e ­A­n­d r­e­A­d ­m­or­e ­g­o t­o t­h­is s­it­E ­h­o­Me t­A­b ­f­or ­m­or­e ­d­et­A­i­l.h­er .Go to. TrendExplorers.com
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

×