From a9177c9ee8d24c981db26c5c68af3aac491e2b5b Mon Sep 17 00:00:00 2001 From: Paul Cowan Date: Tue, 31 Jan 2023 15:12:29 +0000 Subject: [PATCH] run prettier in microsite Signed-off-by: Paul Cowan --- .../2023-01-31-incremental-entity-provider.md | 4 +++- .../{ => 2023-01-31}/catalog-pipeline.png | Bin microsite/blog/assets/2023-01-31/damon.jpg | Bin 0 -> 10263 bytes 3 files changed, 3 insertions(+), 1 deletion(-) rename microsite/blog/assets/{ => 2023-01-31}/catalog-pipeline.png (100%) create mode 100644 microsite/blog/assets/2023-01-31/damon.jpg diff --git a/microsite/blog/2023-01-31-incremental-entity-provider.md b/microsite/blog/2023-01-31-incremental-entity-provider.md index 82fafc3b4f..7737582cbf 100644 --- a/microsite/blog/2023-01-31-incremental-entity-provider.md +++ b/microsite/blog/2023-01-31-incremental-entity-provider.md @@ -8,7 +8,7 @@ authorURL: https://frontside.com/ At the heart of [Backstage](backstage.io) is the [Backstage Software Catalog](https://backstage.io/docs/features/software-catalog/software-catalog-overview), which is a data store that allows an organization to centralize and visualize its many software services and components. Backstage inspects and transforms an organization's disparate software services and parts into a centralized data store. -![catalog pipeline](./assets/catalog-pipeline.png) +![catalog pipeline](./assets/2023-01-31/catalog-pipeline.png) A common use case is for an organization to want to surface ownership and metadata about repositories. Backstage provides a mechanism for discovering and transforming repository information into a standard data structure and persisting it into the Backstage [Catalog](https://backstage.io/docs/features/software-catalog/software-catalog-overview). This process is known as ingestion, where all data is transformed into a standard Backstage data structure known as an entity. Entities in the Catalog’s data store are accessible to the Backstage App via the REST API. @@ -28,6 +28,8 @@ A large organization typically deals with massive datasets. Until recently, inge This is a problem that [DevEx team at HP](http://hp.com) faced when building their software catalog with Backstage. [Damon Kaswell](https://github.com/dekoding), Senior Application Developer on the DevEx team at HP, shared their experience at [BackstageCon 2022](https://www.youtube.com/watch?v=5qHyZntKXRU&list=PLj6h78yzYM2OKySsTuiip3BqmdYZQRnSf&index=13), detailing the problem and the solution that [Frontside](https://frontside.com/) created in collaboration with developers on HP’s DevEx team. +![Damon Kaswell](./assets/2023-01-31/damon.jpg) + The solution HP and [Frontside](https://frontside.com/) arrived at was to implement an incremental entity provider. An incremental entity provider effectively performs a full mutation using a series of delta mutations combined with a mark and sweep mechanism. It paginates through the dataset, tracking entities retrieved from each page and the cursor of the next page, pausing ingestion every few minutes to give the processing loop time to process existing entities. Once it reaches the end of the dataset, it determines which entities were not ingested during this ingestion cycle and emits a delta mutation to delete unmarked entities. Simply by adding a few new tables to the database schema, the incremental ingestion entity provider converts any existing entity provider into an incremental entity provider. These tables allow the incremental entity provider to be long-lived and keep track of its current location in the dataset by persisting a cursor that it uses to page through any large dataset. The larger the dataset, the more pages of data or bursts of work the incremental entity provider will ingest—but there will be no out-of-memory errors, effectively removing scalability problems. diff --git a/microsite/blog/assets/catalog-pipeline.png b/microsite/blog/assets/2023-01-31/catalog-pipeline.png similarity index 100% rename from microsite/blog/assets/catalog-pipeline.png rename to microsite/blog/assets/2023-01-31/catalog-pipeline.png diff --git a/microsite/blog/assets/2023-01-31/damon.jpg b/microsite/blog/assets/2023-01-31/damon.jpg new file mode 100644 index 0000000000000000000000000000000000000000..f1e37d354ab02911e65100bd2aa88509bc5a99fc GIT binary patch literal 10263 zcmbWccUTi$_bxni1rY@4V3b~^S1FGoRgvC>C@4q}kdBl@K#KGtT|ntYKuYLEdN|YrGw29dMD;p5;|3`-y0Kyyq(*NjK z5Zb?ZK{yG$|I?CuCHg-jekJ~|_Rv?7|H{h*Z2-Rq$mzXt@o;f}EF=__*drnSEL>1h?pPZgy&M*G*BK)8KLyK_z zA7=j_UbhImh)GFFNGbmEA|m$wOZ*lo**$S`x`+A{FJ0+5B?7PAeiZ+$vi&-jqydV- z#%+X>ky~no2mP1Yzs&ys5exc%V)kER|A*Hcpb8NG3ld^t5;77J5;Af!0?4nC|HU=R zYyX1sKe+KPZvF+$e}N|qLPQvYl$4Z$@S&x;PDT5F4g3OOSyJO?fg2=5gvCU13s3>>aoM@=TYi@`Orcg$G4pBYlXV|l-R1PI5W}iKdE16b}zFxIiTNmyd7WX zE?GBp%ys2xiHkP(`cWXo13l2r2?;c+<181}&t3r~sZs9f`hLC|v2fdZN5U$)pntAktyGOM z6x@Y*=#kj!UyYm_V{%ooLgz->4OY{ulLf|t3y@qPBF~p$2e+SJkqLG(9@MwRdJ#)gj9msF-tMIQ+rea;Wb%X1xw!LXc~KGF7Kvs!1@Btn0q)fHQ|->mII$mUAKKI_%Pizohyh>sxf4r3vuTa??9f8LpbrrRsK?FE?eAlKAdsj z4k*ggZmwcg@1@s72Dl9~i6${oPI_soRKo8!@ww~;U^6RMq^Y!h#9F@7beqV!5Zow8 zm#=2h&{q4SHu?0zsID&XiDHabNy!=S2W4YSx9lf>RkU_z=dUSOjN`Lfx?i=UQZLt_ z-eMGd9Ap7xsB}Ly)TNoZ(%SEb_vUyuka$2AweS2)sLMt9J_gjAzL;D1#{Jsb7$;~e zk%;xlon1z-n&JG#lp_j(^bQoR;6$m+d2O3MMX(n-TFxiD^Dn*~pKUwr-brJuo4M-? z=h2-Oat;u%oW)i;5EAv^tx_?G@%ZtD}lbep2R?)yvH zaXj<<3wrJbp{r$f#MsuF$y~)Tej=HpvHFTCJYchn2Ra(ijIcj8EqGu=<^=uYEv?XQ z_H+|Y3lWY7c5VOXHvR1j5Psys4_f_^ zjR*dbUQova^%}60b<)i`h7Fb5(##p{zQ^V}w(oLr80o-mP;}^bOLXRhsW&i=k{z~u zOkIySgDmKMTyZrqZVQwCL7$9FNNZY5-4#D#1^mQ_Tive;JpH0oRsx303RZb@6;wl5 zcqoQoNdds9eg2CbKT}=Dlqk)+FR&l}p&Fm`6m%1so}1G0kW!2hmA)g-v;*M zWBwtz%g?aCf{sgK%#h~`b1fTyMolkJm1^F9XF9CrMx!5}-Wq*w8jNvS?fu}HmfXG1 zTj}{U?f&Xv?rM}Xg10FZ$Lm9uAG4@JYn~hzqQ}D`>`Qs*Fs@#h<6KHPky;f>LLKK9 z0cTG#BvE^{_3TJ#`zyq1B96&B*20zDvm~(5IIxL5h@U9`2Sij-(dG-ej$(=T&Og`6 zl6$o;RZi~{X5JksddXY?JJxCOx4B)6XbDKWWc-4aM}w}SyxT5#7jtZr`|zX36~U~* zTdh(N>ue5}!nv^>tve%70c-O{uGV$1FZKlE&WpIkbyLNa@3;c?q?Z&H?SWWT9Lm7( zO-o>n%njF-S{#R(omN>+lt>IeS%%eqw#5-U_Pv|{fqcJ?A4H{FEhDg1iEWp>pP}?) z$~x4C$ifgD$I~qA86BZki8nSUzYfea+k(ANy`$XK@gPaWH(AmtZdfNoeiW z^<38B>=HcrlOKJ5)rNdt;p%xa?Df7RIC;-5|o=^^H@MUJuc~4 zL2WaAOoTojE9TuDxk!yF5z7VM0e^tf_~8Td!n>?)Ej|=4e^eVrC9P%&U?xyI?Tt|i z&#FtiQVi)R1(w!VFL#?o$E4r*o1>)ApT17yc)q3pj$-ox7k_QE)zZgimLf-12P&T? z&X81MhV>VL;m%F&>1*JX*EL@H5HD+iiQQ&R`XPl#tLBnJTX&Z__jP7GFik1<2T}k> zE`3y%uE7HzVck$Z9noj$nhWbD=k3wKL?bQoCCc~Ewed)$=MEqmMWa}r?{A0KEz=S- zFYa5e1HOS)au7kcYOmXwNeI~yWq!;-muw}$%ef;Ruq-d7qFoBM8NFOH*!pd4@_rp2 zh)dpz3wxs!?MGJE4WjkAxxpd2O5ObA^#ih6G+9KV?frwM(~$MAw%cxD<0hhE@CCXt z&Bot2KFDN>7)0Z`i>tH9vcO1CsVB=OABECZ0_H*2B5}1higAenqr46a__|OZpTN6z z6=UBg6cewk@h#svA!_#7YE+H#=eowAeg1u?S6VA+s$4yD@t@B~Ju3*TZE|!wW9jU( zF+~S|Z4LSWV}URl5w`5kCbD-dW7q6B>zMr{eoj|$(G{4T@m}NXrrrrf`jzul;EB03 zA~?OLLgdSZe~R_p7O&a{y{e8yBOuo{I7s2FTswnO13&hs8JP3`Ck&UWbtN>1oy7zr znY!i0uWsdKk+>uVW9JTN7LQ;*-ymY13fRd9V4 z()!Z8jQ?_`P*rxg8}L9&>o)%29ltxaTP{siJ4g22Nhn+GAFV4gf0aDw1dg*A9nxu~ zMhL~O{jqx$659XRb{~re7Ow>J^)=qjU+!(Fgm4a;DezXmuGo@koXq4)bMgs#tejS+ z(^0cQY~tKJVg54WR6ux>C^yrb`Ofw#DO5dKO9rh)0zJ{KFP^9A6|q&T`>m*hMBMnX zF}w5D-^0k=OsB8|7Is`d*h4La2k4t$OOFg@6ifCfs4yVvERG;scO_ueFO{O9snA}{ zT7o&dAY;F_y2oO!Rs2%8V!n!7Wq-oDRXIA#5&w?(Q|3{o5iB@8J> zD{Jci0j0}^WyQQKn_q&Y_k@=U2W}>7t|f?y5hcczlfx*NrNA@2KUo{8`bGBlq|T9s z6iHqyeH6K`JV^2bC|0f1-X3T4ca%!}G*&G)N|BT2=n-;bF!Q*Sj{e6eMnX)CVnB{$ zG`znoBv)Agt+9KCI?%)R>^p>Btc2zJ7Ir-s?QRP*s5sm2#mQT3c73t{hh=eo^sY_; zC21naz$8BOKUvqRv1%jh6UUYf{)0-+S_KU;Qi(|sd$Q5>yO{O^@M{Ne8JJC-f~)Eq z{in(%ify8x`}3|*9^XMX9J~yDKpL$1U7S>r*A%-v;wO@FS#HRWGjtrJDu?s=Vjap!PrZ(_+?Ge|>T6-vbnVeHsEV2<)I6Es3>i{!zO-MhG3)%aXcP+lK?#l{? zQNTR^!qt}Gj*(=;q}QH34?z~#z6llk0}edkY;B40 znhSt84`oCi$9j=~-rBSoE3;z7!VK!bEIYno)8Frq`B{mOh%`fTuU-WU^|JH-+(cCj zw?$^x*j1D|UW8^WH~C2E@N1&QYfVh+fxY?lv{<=cOcCp4g97G~W`3nc0dQ1e``IEX zN;)&T< zPs+r)oj^a}`WvI8D}vyf`}?^cGn!ZC1uk>Pr#8b*3({5AOF=oT7c?ud>Gjd*ydN21 ztxl)`8Ns{yK|FSz{E#0;2M?6UxQ-iFyA&Aemy~mzBYf>1Z=RF{_gFLZVnfPc2g?nNv_sOm->Q3{trj_pL^_?_&tXyWXMe!=4tf zuFPjb;k$|sxmOVaF`aUV&qODP!Qhi}pWlUiuX^`Dh9?^B3iY~VZ>%j#`#C~dZ`ZS* z+XnfWB9;=$aFk-?QILv zK4S4FLI%)1$QkJ`=~n#uvF@!i^Gb#*Gj>NXReo%NRXxZ44=mU@*Ci4s@#h&@ z4>vusjUi22l!`F<#dEHV8;Spc9`^TiL04F5Io}OaA4S$Cg-$0EE5e^+i_RBfVA>am zRhN$^2v!W}lC86!ZQ-gwz?Ey4jqBHYtmEnmiUgU19d>Y5*;@u}fggsHX$viC;~efV znn$eXJ3E5R-VhvL-Oo!f8CxdXN=U`WWhZlChqi|0aVU^mSx*B}Mc{6^!EAXveqHzw@=~!%+^<%amLG}wiOAYu7vckC=z(k(% zqMgH5oOJcWkOg@i9lyNA^Q_HY#YlJsoGu4O;Ov_93x^AKWVHu6WRi8FmU>8TrC*&j zTs1>gAQqwdj^!9CZ5RzCZkPyO+u+d{!lSO&;XPkIBQLMlnp!c82NisGXCs8%t$NDvP3&mSz-fbCy!E6Y!=`n-B5v zY@s_C(=iykvWib)ukVVhkok;xCF5yPFC<9$Z2YWji5e@=ips>r7yn^>8@FKblsijn zBjWiQ!;5 zI`6{ARYvz-$-T1eJ2Tv|GVr%SO_a32yQ3$58n%-e1^#hN1AhS1tG+JVn_5CpTo@ao zD3I2NJ+n*M7jIQM;Du4e^zwdu<3kgQ;FV}~a}E*swH)`%z;9ef-7jDeA2XVSj@=9B zArBsy5hBF%ArxqS*kChrFZL9)qDP4D!!YsDZFXTeJV1jgI*mSNstI$CQ8qp!p<|97SCix@Y`((>3Xgey71}as)G0Vx`3jxp*tF)0@XGak zyn8>&C1Q~WcTI$8iwO@{=OiMtBkVqXi?gVbYd85wtFdWN*S-Y0K`7C6;sIEQEbcZQ zh&j74t2T!7qg1X_UKym*Y`u>`E^-GgZY=WuenA%KLz2&r8AM-aA~AfJt;vN4C_iAi z(7$^`UasOYD9fR=m`xU$9|SrpPj-_0(*5#j-!K33np2f3j+zd*Q^d$L5(9&pqx$@K6wz2VKkN+04ZiyBLAXQdv$`@LWv16jm(+ zztAPG^eL-ZvmXp=v00mT2jM5%5d&@7(M33h3nK#=p&ZNX?4N5Lt(sy~|d{ zDB@}c4+J^mf&P~HRa|gN_JtrN_O4wo!zH1vMmmoN5`7!+z>oD!TtrZ|_{bA?{M4q( zN242V5T_;n3*+ZU>3@>;{h4P&8poscWqpTcxgQg95@UHK$e`?7V*$3{XFL$q?Fp+4 zX~}?r3e#eJu3qUI(UV#3gGO{R5Z}yO)8W|%Mmk4^G4BgxwSW12ei(RGQeEepc3~|4 zM8v6xR(4UUb2K^E)=VXQHjOmfV?w=@&dcJv@8tNYjL?7)@+;*1j{cqZ$3;iPGG`^b zgTtCv+f8*+X|>UDbrzq7wWcS$9{8LLKddo^3u*0vn?Wd@igVeF?8}?hus=mYsJDlX zXNkD$a6G_!L~!gc2N72cWK3@-lQ;3dH`qd`_VjF{#t_a9(U#-$7|s1h!}_w?envWr zU+2rXY((?4ma2M@=cRM z%Rq}q^QH%USbBS$&c-IL^M59nS&15Y1ZyYMdDa&8VY~A03Fi0p+dqqJ z+4vD;r+@C&QxW4cLZe?jc_ORK$sN)Ne>CK(SM$ZQB3GHgpEW~o(I|aK;%<+2I?hAir-F$@^cEXV-V_z)NaekAJ!!75Atpj2Yx>q!=)|AP=6} zBczuIF7zv=3jHau{i0M|{UKw5TU*lmiE&v*gMJ53dR4`z#;ft#jrH3smfwp1I55}1 zx#CNjv?3gvY)5nLR?F_Ef0pZ$$(k_F5K6Br+VV|Ku01HWcmfgSqJQ-2QI6(ZZnMYnKJE#-?BR3*2BVy=JbS)c~d#`pWmtp1yW+g~Qo#KrMYEB`sZs;jt z#$sUhe*ay!BB?ZxD}vt+a$v{(n&O82_Bz7@Y7Z?$qVF}}M~|HF0DqCi)AVrhnaR@R zonZ&ja7e?QvsO)y< zn`6@!3TCj5#}|)U%V32M*3g*|;p zoPplKOL-5|l4tQ8YnZj~66u9wv^}+PGB59nbBdO~nx4tJavgRB(H=WVcC~Rxr}Cj8zyl9PZMUgW2-arqJnafHH zr;Xkh8qs$*$hg5Z-ZgTGJP=>0{&bQseP7;UqGJ(v>X*O@%5U=cTEdjEQtYlZRi_|`{a@J-mwRt^kb zw=!)D+ACwFw2g5a?CjH4>EwCOMJe;^WfjL@qutA`ZZ}54G~GYG^6T)eO@knp{u!fx z(hN^JQ+rRtY23R&)V}(AdD7#K3`SnqR7N2-)RXNRvMFI>;7q3>hHWiIHJ)@xeTve@ zgrhk9=+T){-AZ&}CyWx)i6l?F8Cj}$JqIBWajIMOyz>^-#$46s^9*mTfCr|@{wk=H zz>{G{D{Dl%?^I;@Mx)0iH+_bsCEGiw)tQ=1|Gk4g;=+{EkkNx?KTjmJscJ)pTEhq) zsJsDoAjkX!h&3`pC*xL&8KRlGbwj6c)t3mr^$DZTow&k zqHX2qM`iZAfiSgW**Ne@o_81bkt zcRARIB2Vf2m|!9X`_$-Xk{xY&>M+^N0}DW%z`@i$!8-9KY*yMXaE+qwFN`Mo;d8Cj zZiG+}C#Z!g?De+TnYmwBu*iMtJsDD)R<_P9X0ofHZv~E2=ETyDVi`((Y*%DMD%AGd zIBU*k2Ro&bg01iO=(wA5 zh;n|mI@QnK2|qE+!q-Jdoky}9 z+)bbCxGUEZ`=v>=o;on`rI!zno#YytC9~9?l??OT!~@d4vYi!lf1n)IXqKrL5&I8z z-h5>|j7?XQu+tUB15`!lpD!YYW)-T3%wygW3H#pMD3yE=@Y~Gq#?+|k;$2jpZ)>jO za+w#^g9fQf=I@>YpSwypb(bDnQm^;aOo`m4%1iGaw$XT&6kM7%6`AD5{Q{4#6 zx2n*EHm@P@SyY&d3AklUzMvh62t!UX7t~T0Qxla%Z)QEVS$)H%n3?TY>u#~FVo(1w z(&)xg_h?J-%AJ-K-RrsmNbs%sk_GirvSt-~CZ1%&%MI}aCN^cgt~GiqrO;|rsrH!X z!A*$!{oH(Oh>)ifw*W4#GRIk`;m`GRcCVe;hT&XY)y>BV9x#dddQW+ir=fS2W0E-5 zC$3gAGWrYZs4Gj;1YFqcrL`*@8Dr(Bi-N~>rRei1YX5Vd@JPKMkL2zZxCy3 z1@;#`jVgm=079{#}zqA-kFwncLe7QLtcgf1Scb(z#DzZ9l=SzBpMvb;K(kMfRC zlwHZb1;O#5r@~)MdTN;M(LM1|p61{NwZ4^2NvvC6irP%4VBYz5G)^+ z3({2U%TCEsX`IW>{&vF7xn}Z(Qc1P+>$^O{VwOrJj8G+qs8u!yxo-e9t}j7TW+GsG z^5xUTfRa3jaFk#*oJ;A^uz9ZuadOFeG55hGyzHiEUc$zNqSx9zI@wf3@Yovm#A%>M!;(Mat8 literal 0 HcmV?d00001