Telescope has launched! 🔭🚀

Version 1.0.0 of Telescope, Seneca College’s open-source blog feed aggregator, written by students for students, has now been released! Check it out at https://telescope.cdot.systems/!

Before I reflect on all on the blood, sweat and tears that have been poured into this project since Fall 2019, I would like to quickly follow up on my last blog post, detailing the continuation of the prestissimo development undertaken during our final pre-release sprint.

My Feeds: functionally ready!

During our final sprint, I decided to continue dedicating all of my effort (detailed in my last blog post) to getting the My Feeds page production-ready.

Starting back where we last left off, I tackled two “5 min fix” issues:

Issue #976: move new feed input fields above a long list of existing feeds

Knowing the My Feeds page like the back of my hand at this point, the fix to Issue #976 ended up the simplest “5 min fix” I have thus far contributed to Telescope:

PR #983: involved relocating a single line of myfeeds.js for a better user experience

The second of the two My Feeds-related “5 min fixes” I took care of ended up not actually being directly related to the My Feeds page:

Issue #1002: fix two errors popping up in the console
PR #1006: makes fixes to prevent two JSX-related error messages

Unbeknownst to me at the time, one of the additional fixes I added to it (fixing an ESLint error) ended up resolving an additional issue: Eslint issue in Banner.jsx (#901).

With the two low-hanging fruit issues out of the way, I turned my attention to a real problem that was seriously inhibiting further development of the My Feeds page:

Issue #977: serious input lag was crippling the My Feeds’ page’s form fields

At first, we did not know why it was taking tens of seconds to input a single character into either of the My Feeds page’s two form fields. After some speculation and investigation, we determined that this was occurring because typing into either of the input fields was triggering a rerender of the entire page (including the massive existing feeds list).

(Technical details: typing a character into either of the aforementioned controls triggers a change handler, which updates the component state, which triggers a re-render.)

This was no bueno—I needed to find some way of preventing the existing feeds list component from re-rendering every time the state changes. However, I also needed to ensure that the list would re-render when we want it to: after adding or deleting a new feed.

The solution I came upon is known as memoization. By wrapping the existing feeds list in the React.memo higher-order component (HOC), I could control exactly when to re-render it:

As can be seen in the above code snippet, I wrote a custom comparator function that determines when the memoized component should be re-rendered (when areEqual(...) returns false). As a result, the existing feed list is only re-rendered when a key/value pair (i.e. a feed) is added to or deleted from its feedHash prop.

As a result of getting a handle when the existing feed list is rerender, I was able to ensure that this list was properly updated when a new feed is added or when an existing feed is deleted (which ended up consequently resolving two additional issues: #979 and #909).

I aggregated all of the aforementioned bugfixes (plus a few more) within my final My Feeds pre-release PR:

PR #994: essentially makes the My Feeds page production-ready!

(As a result of my work on this specific issue, I gained a ton of invaluable insight into how React (and the useEffect React Hook) decide when to re-render a component.)

Connecting the final My Feeds puzzle piece 🧩

With the front-end functionality of the My Feeds page completed, there remained only one matter of end-to-end connectivity remaining: ensuring that blog posts from the feeds added via the My Feeds page were displayed on Telescope:

Issue #920: feeds added via My Feeds needed to be queued for processing

Thanks to Telescope’s fantastic set of back-end tools, resolving this issue was as simple as making a small additional to the proper area of the back-end:

As can be seen in the above code snippet, the required changes, here, involved ensuring that Telescope was processing all of the feeds added to Redis (i.e. Feed.all()) in addition to those pulled from the CDOT Wiki’s “Planet CDOT Feed List“.

I included these changes within my final PR of the sprint: #1015.

Peering back in time with Telescope 🌌

In numbers

I would like to extend a huge thank-you to all of Telescope’s ~60 contributors, but especially to its core dev team: @humphd, @agarcia-caicedo, @c3ho, @cindyledev, @Grommers00, @lozinska, @manekenpix, @miggs125, @raygervais and @yatsenko-julia. Each of these devs have likewise documented their own journeys with Telescope, and I strongly recommend checking out their own blogs!

This is not a goodbye. There is always more work to be done in open-source development, and I welcome you to join us!

Telescope: Vamos allá, My Feeds! 🔭👯

Last month, I blogged about adding Telescope’s first Material UI-equipped page: the My Feeds page. Since then, Telescope gone full-tilt with Material UI, and the My Feeds page has grown up considerably!

Today, I will outline the My Feeds page’s latest growth spurt—one that occurred over the course of six issues, four pull requests, and one-and-a-half sprints!

Coming full orbit

As Telescope grows tantalizingly closer to its v1.0 release, I find myself reflecting back on its earlier, simpler days… back to when I was once fixated on the idea of connecting pieces of Telescope together. Since those halcyon days, I have learned much about collaboration.

Critically, I have learned that connecting front-end and back-end components is less of the one-man task I envisioned it being and more of a choreographed docking procedure (or, perhaps, a dance) coordinated between front-end and back-end “teams”.

Fittingly, this appears to be the last dance that I will find myself untangled in before 1.0. (An optimist might consider that meeting one’s goals, and then some.)

The My Feeds Flamenco 💃

I. Allegro

We start our dance off with two relatively-fast and easy modifications to My Feeds:

Issue #914: the My Feeds was not represented in our navigation scheme

Quite simple, really; Telescope was in need of a way to add inform users of this vital page’s existence. My solution was to hyperlink existing text in our site header:

PR #923: the My Feeds page is finally accessible via Telescope’s Header

Speaking of Telescope’s Header: around this point, My Feeds was missing one!

Issue #868: the My Feeds page was missing Telescope’s Header

Unlike with the last issue, the solution to this one ended up being a bit more nuanced:

PR #922: does quite a bit more than effectively adding a Header to the My Feeds page!

We saw and seized an opportunity to considerably improve the structure and styling of each of Telescope’s four pages. Consequently, we said goodbye to the supersized styling seen in the second-to-last screenshot, here.

II. Adagio

Some time during this last sprint, I felt the sudden need to stop and catch my breath. For many days, I simply did not develop (or do much else, for that matter, except gaze listlessly at feeds not belonging to Telescope).

My malaise of the time was compounded by the vertigo of staying sedentary while watching my peers labour more feverishly than ever. Everyone copes differently, I suppose. Eventually, I snapped out of my trance. I’ve since learned that working does help, but in moderation. (Healthier still would be to spend more time reading Telescope; more time connecting with my peers and those close to me, and less time staring pointlessly at bar graphs, trendlines, figures and projections.)

III. Scherzo

The dance now picks up to its formerly-frenetic pace, with three more issues to added to the My Feeds mix:

And, as of this very moment, we have the One PR open to resolve them all:

PR #933: a My Feeds makeover, adding vital functionalities needed of an important page

This pull request represents the most-involved frontend work that I have yet done for Telescope. As a result of this work, I have learned a great deal more about Gatsby, Material UI, ESLint and (especially) React Hooks. I once misunderstood useState() and feared useEffect(). Now, I’m a regular user.

(On a slightly unrelated note: the rigours of building and rebuilding the Gatsby frontend also spurred WSL2’s current RAM-devouring tendencies to leave me without the ability to run Telescope backend using Node. Fortunately, I discovered that running a cache-clearing command after building the frontend deallocated just enough memory for me to get by. Well, what’s the takeaway from this? My next machine is going to be a Linux box.)

IV. Allegro Prestissimo 📍

As our final sprint starts to draw to a close, the dance draws the front and backend teams together as tight as ever. It is finally time to connect the My Feeds page to the backend logic for (i) user-owned feed fetching and (ii) feed deletion. It is also time for the writing of Jest tests, to ensure the logic of our two ends meet seamlessly!

Issue #955: Jest tests are needed to test the new endpoints added to connect front to backend

Having quite a bit of prior experience writing Jest tests from my early-early Telescope work, I did not experience too much difficulty writing these:

PR #958: adds tests for our new GET /user/feeds and DELETE /feeds/:id endpoints

Importantly, having devised these tests, I now know exactly how we will be connecting the My Feeds frontend to the new backend functionalities. As such, I took the liberty of preemptively adding logic to send fetch requests to our to-be GET /user/feeds and DELETE /feeds/:id endpoints. With this connective logic added, I do believe the My Feeds frontend is well-poised. I graciously bow to my partners, awaiting their next move.

More to come as this dance concludes and the code freeze begins. Stay frosty.

Telescope: a pants-off approach (🔭🚫👖)

During our last sprint—titled Goodbye Random Pants—the Telescope team largely focused on slimming down our back end, as well as beautifying our front. That disclosed, please allow me to unpackage my own contributions to these efforts.

No more dateless pin-ups

For quite a few weeks, Telescope was placing one particularly-broken post on a pedestal (atop all other posts) just because it refused to give us a date:

Issue #757 – Telescope was ‘pinning’ a post without a publication date

This infamous post in question was being ‘pinned’ to the top of the Telescope’s feed list due to it being provided with a new date of publication upon being processed. As such, this post was provided with a publication date more recent than any other processed posts, which consequently caused it to be hoisted to the top of Telescope.

The solution that I presented simply involved preventing posts without a publication date from being processed:

PR #823: reject posts without a publication date

Implementing a fix for this issue required adjusting the logic within post.js involving the handling of missing or invalid publication dates. I did this by writing a robust ensureDate() function, which now validates each the two dates associated with each blog post (dates of publication, and of last update):

An ensureDate() function is now used to validate post dates

This function ensures that posts with an invalid publication date do not get processed (via throwing an error), while all other posts do. Notably, this function allows posts that lack a valid date of last update to be processed if they possess a valid publication date to fall back on.

Never drop trou(bleshooting)

Within my last blog post, I described how I “ventured into [four] aspects of Telescope that I had previously had zero interaction”. During this last sprint, I continued my streak, touching three aspects of Telescope completely new to me: authentication (passport.js), GraphQL, and releasing.

Gatsby.js authentication: graphql-passport

As Telescope appears to drift closer toward a serverless future, the team has been slowly enabling our Gatsby.js GraphQL to perform duties that previously required sending a request to one of our many backend REST API routes.

One such key duty was authentication—more specifically, querying user information from our passport.js implementation:

Issue #701: Add authenticated user info to GraphQL via passport.js
The (very-necessary) background reading I was kindly suggested

Although I had previously zero experience with GraphQL, this issue seemed fairly straightforward to resolve: all that needed to be done was (do some reading and then) install an npm package… right?

Well, it so happened that those steps were just the beginning of a lengthy learning experience.

My initial installation/implementation of the graphql-passport module went fairly smoothly. This involved simply involved installing the module, importing the buildContext() function, and adding a context: ({ req, res }) => buildContext({ req, res }) property to the object used to construct our ApolloServer. The only challenge, here, was determining whether or not we needed to pass some User schema to buildContext(), as suggested by some of the author’s code examples (we do not).

Now, with the installation out of the way, a final, crucial step remained: testing it all; ensuring everything actually worked…

…and, it did not (for quite some time).

Within the description of my penultimate pull request (#824), I detailed the exact method that I used to test our graphql-passport implementation:

PR #824: Add graphql-passport to Telescope

Again, in text, here was the method used to test this pull request:

  1. Check out this PR locally.
  2. Run npm install, installing the added graphql-passport module.
  3. Within src/backend/web/graphql/index.js, add context as the third parameter of a GraphQL resolver of your choice.
    • e.g. The parameters of getPosts would be updated as such: (parent, { filter, page, perPage }) -> (parent, { filter, page, perPage }, context)
    • (It is not necessary to make any changes to typeDefs)
  4. Within the function body of your chosen resolver, add a statement to log context.isAuthenticated() and/or context.getUser() to your console.
    • e.g. console.log(context.isAuthenticated(), '\n', context.getUser());
  5. Run docker-compose up redis login elasticsearch (Telescope’s SAML server needs to be running)
  6. Run npm start (the backend must be run locally)
  7. Navigate to http://localhost:3000/auth/login and log in with the credentials user1/user1pass (as per our login documentation)
  8. Navigate to http://localhost:3000/graphql and write a query of choice corresponding to the updated resolver.
    • e.g. a query for getPosts: { getPosts(page: 0, perPage: 1) { id } }
  9. Execute the query.
  10. Observe the statement logged to Telescope’s console, which should look something like this: true, { ... email: 'user1@example.com' }

For quite a long time, Step #10 instead returned false, undefined for me—and I could not, for the life of me, figure out why.

Thankfully, from the start, I have been collaborating on this issue with the extremely-knowledgeable @manekenpix. Unlike the other mystifying issues I’ve dealt with in the past, I did not spend long dwelling on it alone, and soon reached out privately to @manekenpix for a sanity check as well as for “hints” that may push me towards a solution.

@manekenpix was extremely accommodating of my sporadic requests for “clues” in lieu of an outright answer to my issues over the course of a few days. Ultimately, he (and then I, eventually) discovered two problems with backend/web/app.js as it was:

An excerpt of backend/web/app.js, as of commit c1eb65d

The first problem can be observed within this code excerpt: passport.js was being initialized after our GraphQL middleware was being applied. Moving lines 50 to 52 up to before line 35 resolved this particular problem.

However, making this single change resulted in no different output than before: still false, undefined. This problem ended up being extremely difficult to track down, and was only resolved by @manekenpix and he delved past the author’s documentation and into two sample implementations.

Despite receiving a hint that these implementations held the key to resolving the remaining problem, I was still having difficulty singling out the remaining change I needed to make to backend/web/app.js. As such, I requested that @manekenpix publicly share his solution to our two problems:

A comment by @manekenpix within PR #824

As it turns out, the elusive ‘second problem’ that I was experiencing was not, per se, an issue with any of the code that we had added to backend/web/app.js (or its positioning). Rather, it was an issue stemming from how our GraphQL Playground is configured by default.

It turns out that GraphQL Playground, by default, omits cookies sent in response to cross-origin requests. (Of course, this ended up being yet another CORS issue… and, it is worth noting that this default behaviour is the exact opposite of what we may be are accustomed to.)

Configuring our GraphQL Playground to accept cookies (via the playground property shown in @manekenpix‘s comment, shown above) resolved the final problem, allowing me to finally confirm that graphql-passport was working.

With that, I marked my draft pull request as ready for review, adding testing/reviewing instructions to its origin post, and the rest is history.

Now that PR #824 has been merged into Telescope, I plan to utilize our new graphql-passport functionalities within my work during the current sprint (more details to follow within in a subsequent blog post!)

Now released: Telescope v0.8.0 v0.8.1 v0.8.2…

“We don’t speak about 0.8“

Well, perhaps, except right now.

I was fortunate enough to experience Telescope’s releasing workflow firsthand, volunteering to perform a GitHub release for Telescope v0.8.0, officially ending that sprint.

Our releasing process is made very simple thanks to release-it, essentially entailing running a script and answering a few questions. The end result of our release process should result in a fully-fledged GitHub release being added as Telescope’s current release (as was the case with our previous releases).

However, this was not the case for Release 0.8.0: it failed due to a history conflict between my local master branch and the upstream/master branch. Although this conflict was easily resolved by resetting my local master, this failed attempt added a v0.8.0 GitHub tag, necessitating that my subsequent second attempt at releasing specify a new tag, i.e. “0.8.1”.

My second release attempt (v0.8.1) also failed, but in a different and strange way: this time, the release ended up being fully performed on my fork of Telescope, but not on the upstream repository (as before, only the 0.8.1 release tag was added to the upstream).

This result was puzzling, and indicative of a bug with release-it. As such, @humphd opened an issue with release-it, which was very quickly confirmed and patched by their dev team. Not missing a beat, @humphd updated our release-it configuration, which enabled me to finally successfully release Telescope v0.8.2.

“Hold onto your butts…”

…as Samuel L. Jackson put it in Jurassic Park (1993). Telescope is currently experiencing a feature bombardment prior to a code freeze. It’s a rocky time, both here on Earth and up in the cloud.

During this current sprint, I plan to use utilize some of the tools I talked about within this blog post to finish implementing the ‘My Feeds’ page (which I discussed within my last blog post)… details to follow within a subsequent blog post. Stay tuned, and stay safe. 🔭

Telescope: that was not-too-uneasy!🔭

Within my last blog post, I stated:

“From now on, I really do plan to make much better use of our communication tools, including Slack, to seek out help when I need it!”

In that spirit, I am proud to state that I shared three of my greatest headaches during this sprint with our wonderful development team!

All Jests aside—this is the first sprint that I did not focus on tests!—I really am proud to detail what I have accomplished over the last three weeks: collaboratively tackling outstanding issues as well as and developing innovative solutions for both back- and front-end matters.

I am particularly happy about my increased presence within Slack during this iteration (having sent about 200 more messages than I did last sprint), and look forward to showcasing my collaborative troubleshooting efforts I made to overcome several development environment dilemmas I ran into.

Four new frontiers

During this sprint, I ventured into several aspects of Telescope that I had previously had zero interaction with: Redis, SSO and our Gatsby frontend. I also introduced a new technology to Telescope: Material UI. Within this blog post, I will recount my adventures with each!

Frontier one: Redis (and rate limitations)

In order to tackle an outstanding issue—Issue #624: Deal with 429 responses from medium.com, and other feeds—I had to have my first dance with Redis.

Issue #624: Telescope needed a way to handle rate limiting when fetching feeds

However, my initial fumbling attempts to configure my environment led me to inadvertently running Redis locally, as a service, on my virtual machine (which I am no longer using). As a result, my attempts to run a Dockered instance of Redis naturally failed, due to the Redis port being in use by the other instance. After failing various attempts to kill -9 my local Redis process, my plea for help on Slack was swiftly answered by @manekenpix, who was kindly willing to walk me through troubleshooting until we determined exactly what was going on, and what needed to be done!

After I was able to get Redis cooperating, I was helpfully set on task thanks to the introductory Redis walkthrough and issue-specific guidance provided via Slack by @humphd. With this knowledge in hand, I was well-equipped to tackle the issue:

PR #726: addresses rate limitation by delaying processing of rate-limited feeds

Thanks to the coaching I received in configuring my development environment, prerequisite knowledge, and implementation strategy, I was able to implement a solution (with tests) in a pull request.

I owe a very special thanks, here, to @c3ho for carefully reviewing my code and noticing that a piece was missing (perhaps lost during a rebase)… Embarassingly, this was a ‘connecting piece’ of code, and ‘connecting things’ was a stated interest of mine… No harm done; I added it within a second pull request. (Speaking of connections, I may be repeating myself, here, but I really am extremely grateful for the connections our team has forged, that allow us to collaborate so closely and effectively.)

Frontiers two and three: (reviewing) SSO and Gatsby

Before I chat about my next assigned issue / pull request, I want to take a moment to talk about a significant pull request review that I undertook.

During our last sprint, @Grommers00 went out of his way to review my OPML-related changes—willingly delving into an unfamiliar domain of Jest tests and backend routes. When he needed someone to review his Gatsby login functionality, I decided to return the favor and venture into his domain; a technology that sends shivers through many of the development team at the mention of its name: SSO.

What I set out to review – PR #672: connecting to our Gatsby frontend to our SSO

Of course, before I could begin to test the login component, I needed to first answer an important question: “so… how do I run the Gatsby frontend?” Not only had I never touched any of the SSO functionalities, but I had never even started the Gatsby development server before. I knew I was a bit out of my depth, here, and shamelessly leaned heavily on the assistance of @Grommers00, @humphd and @manekenpix in getting my development environment set up.

Of course, once I learned how to run the Gatsby development server in tandem with the backend, I also learned that doing so completely maxed out my virtual machine. It was time to ditch the VM, and, thanks to @humphd, doing so was easier than ever:

Saying ‘goodbye’ to my VM was as easy as that! (Well, almost…)

Once I was VM-free and able to run the frontend locally, during my review of the login functionality, I uncovered a redirection-related bug. Once this was filed as a separate issue, I was able to complete my review! It was very cool to witness some of the workings of Seneca’s SSO functionality (although, admittedly, most of it still goes over my head).

Frontier four: Material UI

Two weeks ago, a discussion was held regarding choosing a React library for use as a UI toolkit. Around this same time, @yatsenko-julia was scheduled to deliver a sketch/mockup of the ‘Add Feed’ component—a functionality that myself, @Grommers00 and @c3ho were tasked to deliver during this sprint:

Issue #725: a frontend component for our ‘Add Feed’ functionality was needed

Once the aforementioned discussion appeared to favor of Material UI, I decided to cement this choice (as well as contribute to the aforementioned task) by introducing a prototype of an ‘Add Feed’ component that was based on @yatsenko-julia‘s work but fully implemented using Material UI components:

PR #735: adds a /myfeeds page wireframed with Material UI components

Simply running the Gatsby development server was one thing, but writing a component for it was an entirely different beast—especially as the first non-index page, and with obviously zero pre-existing components to base component structure off of. (As a silver lining, I learned that Gatsby can automatically implement routing for Gatsby Pages, which I took full advantage of…) Nonetheless, I did prevail in the end, and would like to think that my work helped pave the way for the all of the Material UI-based components that are now beginning to populate Telescope!

One of my favorite things about contributing to Telescope is the opportunity to gain hands-on experience with very new technologies. I was always amazed when I observed contributors just plugging in new libraries and tools into Telescope and seeing those tools grow to become essential parts of the project. I am very satisfied that I received my own opportunity to do just that with Material UI.

Hello and goodbye, Docker Toolbox

After my prototype of the ‘My Feeds’ component landed, @c3ho took on the task of connecting it to the backend. During the process of reviewing his PR, I, once again, ran into issues that I suspected stemmed from my environment setup, specifically regarding Docker. Previously, on my virtual machine, I had no problems running Telescope’s backend via Docker. However, on my laptop (which runs Windows 10 Home), without my VM, I am forced to utilize legacy software (Docker Toolbox) to run Docker. Thanks to this software, I encountered a unique issue preventing me from working: apparent communication issues between my Redis and Telescope Docker containers. After exchanging over one hundred messages between @Grommers00 and @manekenpix in an attempt to troubleshoot this puzzling issue, we decided that it would be best if I ran Redis locally (via WSL) instead of via Docker. Since moving away from Docker Toolbox, I have yet to experience any more issues with my development environment (fingers crossed)!

5-min fix: for the sake of emoji-sized emojis

I suppose I should quickly touch on a quick styling fix I added to the frontend. Knowing that it would be prime time for Telescope reading soon, I wanted to help @lozinska get our blog posts looking fresh (and proportial). I followed up on her pull request with a quick one of my own. I hope that this little 5-min fix saves more than five minutes of painful reading, as emojis should no longer take up one’s entire screen!

From collaboration to delegation

In addition to the collaborative efforts that I’ve touched on throughout this post, I wish to also briefly outline an experience where I was approached and requested to delegate Telescope work—to determine reasonably-sized work for a contributor for the current sprint, and to help guide them through its completion (much like what @humphd does for the other members of the Telescope team).

Shortly after I adapted their wireframe for the ‘Add Feed’ page into a React component, I was contacted by @yatsenko-julia, who requested that I write up an issue to substantiate their contribution to the current sprint—an issue that pertained to the newly-implemented ‘My Feeds’ component.

Not wanting to open an issue that could conflict with @c3ho‘s then-in-progress work on updating the component’s behaviour, I decided that it would be most appropriate to come up with an issue related to the component’s styling. I was then reminded on an ‘annoyance’ that I had observed during my work on the component (and even noted as a footnote in my pull request for the ‘My Feeds’ prototype):

I took this little annoyance of a styling qualm, investigated it, and wrote it into a new issue (alongside a secondary annoyance that I subsequently discovered):

Issue #741: exempt the ‘My Feeds’ component from two overbearing styling rules

This is certainly not a trivial issue (and is one that I myself do not know how to immediately solve). Complicating the completion of this issue is, of course, the prerequisite environment setup (one that I have just finished documenting my struggles with!) as well as the need to familiarize one’s self with the technologies and components/stylesheets involved.

I have and will continue to make efforts to lend my support in helping @yatsenko-julia resolving this issue (including differentiating the strange quasi-frontend—which @cindyledev observed, here—being served locally on port 3000 versus the hot-reloading Gatsby development server that serves on port 8000… very strange…).

Setting up the Gatsby development configuration can be tough… take it from me!

Overall, I am glad I had a chance to delegate some work during this sprint. Project and community management are interests of mine, and I value every opportunity to practice them while working on Telescope.

The next frontier?

Currently sitting in our short stack of open pull request lies one of my own, one that we determined to be blocked pending the implementation of ‘feed ownership’ functionality:

One lonely pull request colorfully stands out from the rest

This blocked PR was opened in an attempt to address an ancient issue that I had signed up for during a previous sprint—Issue 294: Keep feeds in Redis synchronized with wiki over time.

Issue #294: Redis feed set should always equal wiki feeds plus other feed datastores

In order to fully address this issue, each feed added to Telescope (i.e. via the ‘My Feeds’ component we implemented) that is not in the wiki needs to be first associated with additional metadata, including information associating that feed with the Telescope user that added it.

Tackling this issue will present another challenging Redis-related task that I am looking forward to chewing on in the weeks to come. Stay tuned (and check out my status report presentation for a preview)!

Telescope: if you bite off more than you can chew, share it with the dogs (🐶🥫)

For the past several months, I have helped progress the development of Telescope, Seneca College’s new open source blog feed aggregator. Until now, my contributions have mainly been things I could (and did) do alone: fixing small bugs and implementing minor feature requests.

Two weeks ago, I set two new goals for myself: to take on larger issues facing Telescope, and to collaborate more closely with my fellow Telescope contributors on such issues.

What I bit off (🥫🥫🥫🥫🥫)

I had originally planned to take on and resolve three issues by this time—each at least as large as any issue I had previously taken on:

  • Issue #595: Write Jest tests for src/backend/web/routes/opml.js
  • Issue #294: Keep feeds in Redis synchronized with wiki over time
  • Issue #624: Deal with 429 responses from medium.com, and other feeds

It’s perhaps also worth mentioning two issues that I intended to work on but that ended up being inadvertently resolved before I had a chance to: #562 (which I ended up resolving in my PR #550) and #608 (which @humphd ended up resolving in his PR #618).

What I managed to chew (🥫)

My previous work for Telescope involved completing the implementation of Telescope’s API endpoint (currently /feed/opml) that shares its aggregated feeds as an OPML file (PR #394).

However, at that time, no tests had been written for that endpoint. As I had prior experience writing tests for the first iteration of Telescope’s inactive blog filter, I felt quite confident and well-equipped to tackle this issue (#595) on my own. So, I did so, resolving it with a PR (#644).

What I could not chew (🥫🥫🥫🥫)

Issues #294 and #624 are decidedly outside of my current zone of comfort and experience (e.g. with Redis)—ideal for collaborating on with other Telescope developers. Unfortunately, I was not able to address either issue fully in the time I had.

Over the next two weeks, I plan to prioritize these two outstanding issues, and to put our Slack channel to good use in order to communicate and collaborate on each issue.

What I chewed instead (🥫🥫🥫)

During our latest sprint, I wanted to contribute more than a Jest test file for the OPML endpoint, so I piled on a larger (but manageable) issue into my food bowl:

Issue #638: Add tests for /feed/* routes

This issue was a logical extension of the work I had just completed, but on a larger scale: I was to write Jest test cases for not one additional backend route but four!

How (not) to yelp for help 🐶

I had relatively little difficulty writing the tests for our RSS, ATOM and JSON feed endpoints. However, that changed when I attempted to test the structure of the data output by our /feed/wiki endpoint: my test seemed simply unable to fetch the data (whereas I had no issues doing so using our live URL)!

I was simply stumped by this issue, and assumed some complex underlying issue, so I decided to package a plea for assistance into the slides I prepared for our weekly progress report, hoping to solicit assistance as a part of my presentation. However, the feedback I received then made me realize that I should have simply brought up the issue in Slack, and it would have been resolved in minutes!

Sure enough, shortly after I brought up the issue in Slack, I received the help I needed. I was then able to complete the structural testing for our /feed/wiki endpoint and open a pull request for all of the aforementioned work (PR #670).

From now on, I really do plan to make much better use of our communication tools, including Slack, to seek out help when I need it!

Lending a paw 🐶

Over the last two weeks, despite my admitted shortcomings in asking for help, I have still definitely progressed with my goal of upping my collaborative efforts, mainly by reaching out to others who need help (and, in turn, being reached out to!)

During the final hours of our last sprint, I took it onto myself to review our Gatsby frontend’s login functionality. This was my first time building and interacting with our new Gatby frontend, and I naturally ran into quite a bit of Gatby-related issues that I finally resolved thanks the help of the PR’s author, @Grommers00 (who also kindly sat down with me earlier to seek help reviewing my PR #644).

Furthermore, in the process of completing the review of the login code, I wrestled quite a bit with terminating a rogue Redis instance (which I later learned was running as a service on my VM; thank you to @manekenpix for walking me through the troubleshooting process for this)!

When last Monday’s triage meeting found itself short a leader, I volunteered to facilitate it, and continued to do so even after that leader arrived. I enjoyed that experience a lot more than I thought I would, and look forward to leading this next triage meeting, as well, alongside the aforementioned leader who subsequently volunteered to help me out to return the favor.

What’s cooking now? 🥘

During our current sprint, I plan to continue to take on larger issues; to collaborate more efficiently; to continue to take Telescope a little bit closer towards that highly-appetizing Release 1.0!

Telescope: connecting the (c)dots 🔭

  Over the last few weeks, I worked on a myriad number of issues to address my goal of becoming more familiar with multiple areas of Telescope:

https://lh4.googleusercontent.com/EyXbgL54Am6FVzHI0k0jIrcn0pv5CR-7UroZXhCymfoicgdTRbhNm0eNp5QET1IyLPXSYasx89dGF4f-Zgtvu1KKS9vMYe9HSqntUL4KJiQwDkQSzVhlUJBVjle1mp2HKn-c6m1fRLo
https://github.com/Seneca-CDOT/telescope/projects/6
What I’ve been working on
https://lh6.googleusercontent.com/ED4sRwEAJb51gxvt0gWBlJwudhAQU0BV4tDvp_fxjANog6bM6pQ-poA4fYuJd_MXVMS2lqLN86E5mAiO2XqtsCY0Un47IHmiln8Za_z_XCeerbPax1U1_0KpE9qRWUDpIal_sxqcbNc
I examined Calvin’s issue and judged no further tests were necessary.
https://lh3.googleusercontent.com/_1d4lVGTrJB41fZIQkfuIw0IVdARoeQXUR6ijF_nMFhqlvzGpoGBWpiQTSYRepaXlBv7XIPj4mgz-ENhS_DaBLCgLaYT0L4UQp7hJYTvPLYes7mfB2HQF29WdO7MVUARxS43W1Xr0EI
https://github.com/Seneca-CDOT/telescope/pull/553
I have been adding several logging statements to Telescope; here’s what they look like…
https://lh6.googleusercontent.com/CnzfT-Aqx9N_t_kWRjwwU0-NqYHUabZydvG_QCIeqgtLkFRO85rJZXcmoQJU9kAeBKEnzbWC3YnMuDESdIgbYtIo2QVYESu5qmhTVb7njwxj-5Avk9A5DydU-1_LqDtaY7jW8Cq4y-4
https://github.com/Seneca-CDOT/telescope/pull/550
… And here’s how they’re implemented.
https://lh3.googleusercontent.com/x7iOG7Iyn0vuNNmFReeitlgfEssEVMVSpEuws6RfCUFdyYKkAG5j1KEQjE6YOtYNsAbLKDvXExUpL53wRZQYzhLEMlnvY_J6v5Va9j_cCniojcCP4nS270K3PzGJurhjYa6MxtzSIQI
My work on the OPML feed packager is in progress; modules have been ‘plugged in’. PR soon!
https://lh4.googleusercontent.com/FmHW44eya8x6c_XN0Znw97o4IR_utk3y0tI02AcwcVUXhJyqjorRYT6rmtHdooHFZtqdZMmIRpOeQaKcilxdPLIt_kCaXWV5Yj0aHYhKimU7mfP7Al948oVCHVHxKpDLc7vocPwmCOk
https://www.moves-tangier.com/
I met with Sofie, the project lead of MOVES, an organization for volunteers working in Morocco, about open-sourcing the development of their to-be web app. Exciting times ahead!
https://lh6.googleusercontent.com/DTvzsvz0H81NxB7L_8VdQb-E7yu7LDYUci_rPoW1_4lhrfjSjwhppHbCEbGcojLFAwM0FUDexCcNrno_IhSO53_GILZ5JjJTfVW9hIZw_BzgMwPtD-RQ1eaffDcv440yc9dYkT-5Krk
https://layer5.io/blog
My blog posts are poppin’ up on Layer5’s website!

Telescope: post-solstice planning 🔭

During the past few months, I wrote Jest tests for (and re-implemented) Telescope’s preliminary inactive blog filtering functionality (which, at that point, used file-based storage. Currently, this functionality is in the process of being upgraded to utilize Redis by my peer Calvin).

As such, so far, my efforts have been concentrated within a single one of Telescope’s several ‘components’ (namely the “pipeline” that handles/validates parsed blog posts). As a result, I have yet to develop a strong “bird’s eye view” of Telescope as a whole.

Over the course of the next few months, a personal goal of mine is improve my “bird’s eye view” of Telescope by expanding the scope of my contributions to include more of Telescope’s ‘components’.

I plan to achieve this personal goal of mine (as well as usefully aid Telescope’s progress towards production) by tackling deemed-important issues that not only the involve the implementation of individual components, but with how each component connects to each another (component ‘connections’).

Specifically, over the course of the next few months, I plan to tackle this goal in two parts:

Part 1: Aid the triage of component connections, e.g. via Labels: “connection: missing”, “connection: unfinished”, or “connection: improveable”.

Part 2: Aid work on “unfinished” (and, subsequently, “improveable”) component connections.

To start, over the next two weeks, I plan to take the first steps needs to complete the first part of my plan:

Step 1a: Compile a list of components and connections shown in Telescope’s ‘chalkboard diagram’ (as well as those identified during meetings, etc.)

Step 1b: Compile a list of every file within Telescope’s codebase that (should) house each listed connection.

Step 1c: Begin triage by determining which listed files are blatantly devoid of an expected connection (e.g. missing expected import statements, etc.) as “connection: missing” (e.g. by opening a new Issue, or labelling relevant existing Issues).

In the two weeks following the completion of these first three steps, I plan to take on the remaining steps of the first part of my plan:

Step 1d: Analyze the remaining listed files (i.e. those that are not blatantly ‘disconnected’) to determine which contain connections that are all working as expected, and triaging these files as “connection: working”. Working connections that appear in need of nice-to-have improvements should be instead triaged as “connection: improveable”. The remaining files can therefore be triaged as “connection: unfinished”.

Step 1e: For each file triaged as “connection: unfinished” (e.g. unused import statements, incomplete logic, etc.), file an Issue that describes (i) what appears to have been done to begin implementing this connection and (ii) what appears to still need doing. (If some non-obvious bug is preventing the success of this connection, this Issue should also be labelled as “type: bug”.)

In the weeks following the completion of the first part of my plan, I will tackle the second part. Specifically, I will help resolve the Issues that I opened during Step 1e.

Throughout each step of this plan, I hope to continually count on the advice and aid of my fellow Telescope contributors—my instructor and peers—who I know to each possess a keen “bird’s eye view” of Telescope.

(As an aside, I am also particularly interested in helping brainstorm ways to make better use of GitHub’s project management tools during the weekly Telescope Triage Meetings the Telescope team is planning to hold.)

Telescope: the sky is the limit! 🔭

In a recent blog post, I documented my initial contributions to Seneca College’s latest open source blog tracking tool: Telescope. Today, I am proud to showcase the latest contributions that I have since made to Telescope!

With Telescope, my main focus has been developing unit tests for one particular functionality: the inactive blog filter (“IBF”). During my initial round of contributions, I addressed the issue of IBF unit testing by creating inactive-blog-filter.test.js. At that time, this test file only encompassed unit testing for one of the two functions defined in inactive-blog-filter.js — check():

My summary of my pull request, #211.

Trouble on the horizon

A short while after my pull request was merged, I stumbled upon a rather troubling Issue:

Issue #268: inactive-blog-filter.js is shown to have overall weakest test coverage!

It turns out that—although I had written tests for half of its functions—the tests that I wrote for inactive-blog-filter.test.js were covering a much smaller fraction of inactive-blog-filter.js than that… I had no idea it was that bad!

It became clear that, in addition to writing tests for the remaining function, I needed to seriously boost my test coverage!

Reorienting…

It turns out that Telescope’s development environment had evolved quite a bit since I made my last pull request. When npm test just refused to run (after all sorts of npm install attempts, I realized that I was going to have to ask for help.

Naturally, I first went to check out the Telescope Contributor’s Guide. However, after this simply led to more struggling, I discovered an issue (#386) filed to rewrite/clarify the current steps needed to set up the development environment. When I noticed that this issue was assigned to my peer David (@drwm-base), I knew exactly who to ask for help. I promptly reached out to him via email.

Corresponding and collaborating!

Not only did I receive the help I needed through my correspondence with David, but I was able to return the favor by helping review and pass his pull request (#398), which resolved the issue that I ran into for good!

Half an hour after the issue concerning IBF testing (#197) was posted, the original author of the IBF posted a second issue:

Issue #199: reimplementing the IBF’s update() function to work asynchronously

A week later, while I was still plotting my approach to testing update(), my peer Timothy (@MeisterRed) took charge of reimplementing the function in question.

At once, I was left stuck with a thought: will all my work be for nothing? Will all of the tests I write for the current version of update() be immediately invalidated by its impending reimplementation?

The answer soon came to me, as I eventually realized that it did not have to be that way; that our work didn’t have to conflict. Instead, we could simply choose to collaborate! I struck up a conversation with Timothy (via Discord), where we got on the same page regarding the intersection of our assigned tasks, and brainstormed up an implementation strategy together!

It turns out that I was not the only one interested in writing tests for the inactive blog filter! My peer Paul (@ImmutableBox) posted a comment within issue #197, requesting my approval of his addition of unit testing for a helper function (which encapsulates some of the date/time calculation logic used during the updating process).

Of course, I was grateful for the help (as I had totally overlooked testing that function in the past!) I helped review and land Paul’s subsequent pull request and, just like that, coverage for the inactive blog filter began its ascent!

A back door into a black box

Implementing unit testing for the inactive blog filter’s update() function ended up proving quite the challenge. This function reads to and writes from files that are critical to Telescope’s operations: it should go without saying that modifying these files in order to test update() is a really bad idea for many reasons (including because many contributors still enjoy to git add .)

So, modifying the I/O files were out of the question. How about passing parameters to update(), or evaluating its return value? Unfortunately, the function signature of update() is just that: is has none of the above.

Jerry, the original author of update(), referred to it as ‘black-boxed’, and I was starting to realize what he meant. How was I supposed to extend test coverage to lines of code that were completely inaccessible?

I decided that the way to go here would be using optional parameters: update() would still be able to be called and function exactly as originally implemented, but its unit tests could now force it to read from and output from a set of test files I designed.

However, I knew that I also needed to take great care when handling those files, which the tester programmatically generated: care was needed to ensure their removal after the tester finished, stopped, crashed, etc. Otherwise, the result could be random test files of indeterminate state being littered throughout the working directory of various contributors (who still love to git add . — a whole lot of headaches!

Testing tests for tests

After I had written all of the test logic, I encountered a limitation that was preventing my tests from passing npm test: time.

The update() function utilizes Telescope’s feed parser to fetch and parse any real RSS feeds associated with the test data. Parsing a single feed takes an entire second, and each call to update() invokes the feed parser many times, resulting in my tests ending (failing) way before most of the test feeds finished parsing. I needed a way to parse feeds orders of magnitude faster.

My solution was to write my own mock feed parser (which simply returned preparsed feeds given their URL). After all, testing the feed parser lies outside of the purpose of inactive-blog-filter.test.js.

Finally, it was completed. Everything was async; npm test was passing (locally); the new coverage was green green across the board… So, I opened a pull request…

Pull request #395: resolves issues #197 and #199, and largely addresses #268
…but not at first!

…and, despite my local CI tests succeeding, the remote tests promptly failed, with limited indication as to why. With very little initial information to work with to hunt down the cause of this disparity, I decided to gather more information by laying a minefield of logging statements throughout my code.

Bingo. An environment variable, though defined locally (by .env), remained undefined during CI testing. After inquiring about this on Slack, I learned that this is expected behaviour (no .env file on CI), which must be accounted for. I also accepted the task of filing an issue to ensure that all other calls to environment variables would likewise recieve proper handling:

Issue #396: ensure codebase accounts for lack of environment variables on CI

The stars align

I commited an adjustment to my code to ensure a default value for the called environment value, and, as anticipated, it passed the remote CI testing!

Shortly thereafter, my decision to collaborate on my task was further rewarded: two of the peers previously mentioned in this blog post took the time to thoroughly review my lengthy (and occasionally complex) contributions!

At this time, I’d like to pointed out that, for what it’s worth, inactive-blog-filter.test.js is currently Telescope’s largest test file at 5.45 KB — almost an entire kilobyte larger than the second-largest! I am extremely grateful for the swift and thoroughness of my peers in reviewing my pull request!

The test coverage of inactive-blog-filter.js is now all in the green!

Oh, no(de)!

Soon after my pull request (#395) was merged, an issue (#406) featuring an error thrown by my code was posted:

Issue #406: an inactive blog filter test function was mysteriously failing

After investigating the issue further in person, it was determined that this was a dependency issue: my tests utilize the fs.promises API, which is supplied only in relatively-recent version of Node. Phew, I was a bit worried that I had somehow introduced some subtle bug into the codebase!

I sought to prevent this issue from being reencountered by specifying updating our package.json file to specify this Node version requirement.

Pull request #412: specifies the Node version required by inactive-blog-filter.test.ts

And that brings us to today.

I’ll promptly end this sprawling blog post by simply saying that I greatly enjoyed participating in this new, exciting collaborative project and can’t wait to read my blog posts on it.

If you haven’t already, be sure to check out Telescope on GitHub.
Interested in contributing? Check out the Telescope Contributor’s Guide.

(To correct the title of this blog post: telescopes rarely limit themselves to the sky; I have learned that there is simply no limit to our combined efforts. — Rafi)

Layer5: Landscape Spectrums Revisited

Several weeks ago, I wrote a blog post about introducing “comparative spectrum” tools to the Layer5 landscape—a webpage which I have transformed multiple times, now!

Up until now, the contents of each blog post I have written were centered upon a freshly-opened pull request. Shortly before writing my last post, I did open one such pull request (layer5#238). However, since my PR remains open, and since I am still working on the same issue my PR addresses, I have opted to instead push a new commit to the aformentioned (but, now, renamed) PR instead of creating a whole new one. This newest commit introduces a major step towards resolving the issue at hand, and will be the subject of this blog post!

The issue at hand: layer5#211 – add comparative spectrums to the Landscape page

Note: this blog post will frequently reference the update and documentation comment that I made to accompany my last commit to my PR. I consider this hefty comment to be an extension of this blog post, so I strongly recommend giving it a read!

My initial commit towards introducing comparative spectrum tooling did not represent a full solution. Thinking about what might constitute a full solution for the issue at hand, I came up with three criteria:

What was still needed:

1. A complete toolset

Within Issue #211, Layer5 lead developer Lee indicates four slides upon which comparative spectrum graphics may be based upon. During my initial commit, I created graphics that emulated those found within the first three of those four slides, but I had yet to implement the graphic found in the fourth: a table. I was also less-than-satisfied with the ‘grid’ design I had implemented in response to slide 7.

2. A modular design

The designs I am implementing are meant to be used to present opinionated information. As I am not nearly knowledgeable to form opinionated comparisons between service meshes and related technologies, I have instead made my goal to make it as easy as possible for other Layer5 contributors to utilize the graphics (“[comparative] spectrums”) that I am designing. Doing so would require encapsulating (templating) the spectrums I design (new and old), within their own HTML files, and providing a means to easily invoke (and customize) instances of those templates into the Landscape page.

3. Documentation

Because the templates I design are intended to be implemented and customized by other contributors, it is especially critical that those contributors have access to complete and thorough documentation detailing the intended use of each templated design.

What I delivered:

1. A complete toolset

As documented in my update comment, I designed a table that very closely resembles that of Lee’s slides. Implementing the table was interesting, as it ended up requiring quite a few nested Liquid for-loops. Also as documented, I revamped the grid-based tooling and am quite pleased with the improvements made. The end result of these efforts is a complete set of comparative spectrums.

2, 3. A modular design and documentation

The bulk of the documentation I wrote details how to deal with the modular solution that I designed and implemented to enable comparative spectrums to be included into the Landscape page (through the use of Liquid include tags). The documentation speaks for itself; check it out!

What’s next?

I would not have considered the documentation I delivered to be complete if I did not report on the issues still present in the work I delivered. I denoted these as ‘Current issues’ in the documentation, and they represent work that still remains to be done. I know that I can count on the Layer5 community to assist me with small issues like these, and, once this pull request receives some forward motion, I hope to start zeroing in on its smaller issues. Fortunately, I have made plans to continue working with open source well into the new year!

Introducing Comparative Spectrums to the Layer5 Landscape

During Hacktoberfest 2019, I worked on the website of Layer5, the service mesh community. More specifically, my contributions concerned one specific page of that website: its Service Mesh Landscape page.

Shortly after making these contributions, I was graciously welcomed into the Layer5 community by its lead developer Lee Calcote, who later directly approached me to assist him in furthering his vision for the Landscape page by implementing “comparative spectrums”. Wishing to focus on my remaining two Hacktoberfest contributions at the time, I suggested surveying the interest of other potential contributors for this project. As such, an informative issue was opened:

Issue layer5#211: add comparative spectrums to the Landscape page

A month later, I fortunately found time enough to make some major progress into this issue.

This issue is quite highly conceptual, much moreso than any other issue I have thus far worked on. I decided that the best way to approach this issue would be to focus on establishing a set of tools that could be used by more-knowledgeable contributors to later construct comparative spectrums—opinionated data visualizations. As such, my first step was to build a set of responsive graphical elements that resembled those provided as examples. These include:

(1) Sleek information cards:

or this:

(Source: ONIF Container Networking Panel, slides 5 and 6)

(2) Gradiented, labelable double-arrow banners:

(Source: ONIF Container Networking Panel, slide 5)

(3) Graphically-augmented tables:

(Source: ONIF Container Networking Panel, slide 8)

(4) Spectrum-like graphical comparisons:

(Source: ONIF Container Networking Panel, slide 7)

(1) Although there may exist libraries that may help construct such graphical elements, I did not want to introduce additional libraries to accomplish my work. Instead, I wanted to make good use of Layer5io’s preexisting Materialize library. Fortunately, Materialize strongly supports the design of card-like structures, allowing me to more easily implement some.

(2) Unfortunately, it does not seem that Materialize supports the creation of graphical arrows, so I just ended up styling my own.

(3) These tables can be quite easily implemented and styled with or without the use of <table> tags. I anticipate these will be given a proper fleshing out after the other graphical elements are further developed.

(4) These unique spectrum graphics present the most challenging, of all the comparison tools, to implement. Snaking spectrum bar aside, I needed to develop a responsive solution that (a) allowed contributors to programmatically place image thumbnails at preset horizontal positions, while (b) ensuring those thumbnails placed at the same position do not overlap with one another (and instead align vertically atop each other).

This presented a problem to solve: I required a means to ‘float’ elements toward a horizontal line; to ‘stack’ thumbnails atop one another, without overlap, if they are given the same horizontal position. After researching quite deeply into potential solutions to this problem, I ended up stumbling across some CSS property-value pairings that I had never heard of before: display: grid and grid-auto-flow: row dense. Miraculously, I discovered that these could actually function as a basis for a solution.

After styling the graphical elements, I continued the website’s trend of using Liquid tags (e.g. for) and YAML lists to help modularize my solutions, which should aid future contributors in generalizing them.

I opened a pull request to share my work as well as invite collaboration:

Pull request layer5#238: description of tooling introduced
Pull request layer5#238: example images of tooling introduced

The pull request was quite well-recieved:

Comment by Layer5 team lead on pull request layer5#238

As I mentioned in a recent Layer5 community meeting, I am very excited to continue working with Layer5; to refine and expand upon this set of new, visionary tooling for their landscape!