Category Archives: bluesystems

Discover more in 2017

With 2017 starting, we’re getting ready for the next Plasma 5.9 release and with it a new Discover release.

This will be a special release for two main reasons: further add-ons integration and Kirigami.

New Stuff

One thing pending for a long time was to actually better integrate the different parts of the system that can be integrated. To do so, Discover now will automatically import all knsrc files present on the system and offer them as categories.
We’ll be able to go beyond Plasmoids and Comics (!) and make it possible to explore: Plasma Look and Feel themes, cursors, icon themes, window manager add-ons and different application-specific resources.

This won’t have a big impact on Discover’s performance because the backend code was refactored to be able to have several backends loaded even without knowing all of the resources available. Queries are now asynchronous and parallel.

Discover KNS Addons

Kirigami

Kirigami was adopted in the previous release already, and this release will use Kirigami 2, the port to which did not require much work.

The most significant improvement is the inclusion of keyboard navigation, which was requested by several people and is now finally available. Do you hate it when you’re required to use the mouse? Rejoice!

Extra: Snappy support

It’s 2017, not all of the applications are coming from your distribution anymore. To start getting things in place for different software distribution sources I started working on a Snap backend, which allows us to manage applications coming from this system.
Some work will still be required and it won’t be included by default, since it doesn’t yet support AppStream (although we discussed it and it seems it will happen soon), but if you’re curious feel free to take a look and give your feedback!

GNU/Linux bundled application ramblings

It’s impressive how in the last few months (and especially the last few weeks) the discussion around bundled applications for the GNU/Linux Desktop has sparked.

It’s especially interesting because:

  • The problem is not new.
  • The solutions that have attempted to tackle the problem in the past have been ignored (both by us developers and by distributions).

The TLDR

First, let me try to subjectively summarize the problem: Historically, the resources we get in GNU/Linux come from the distributions. Anything: executables, libraries, icons, wallpapers, etc. There’s been alternatives to all of those, but none has flourished as a globally adopted solution.

This guarantees that everyone using a distribution will have access to the resources the distribution can offer. The more powerful the distribution is, the more we get. There’s limitations nevertheless, so some restrictions have to get in place. The ensemble of limitations and technologies adopted will effectively define the user’s experience.

This works. It has worked for years and, given the technology is in place, it could easily keep working. Like in most engineering solutions there’s drawbacks and properly addressing them can bear some goodness. It seems like now it’s the moment to review this situation. Let’s enumerate some of the problems we have nowadays:

  • We have users using really old versions of our software with issues we’ve solved in versions they can’t use.
  • It’s really hard for GNU/Linux users to get users to test unstable versions of our software.
  • We have users who want to use fresh versions of some software but not in the whole system.

There’s been many solutions to fix those, some easily come to mind: ArchLinux’s AUR (with yaourt), Ubuntu’s PPAs, big-tar application packages, OpenSuse’s OBS, and possibly others.

Far from showing the maturity of the Linux desktop, what this depicts is the deep fragmentation we’re into: we have come up with different solutions that break the established distribution paradigm by lowering the restrictions and considering the resources offered as unsupported (often tainting the whole system).

What has appeared recently is sandboxing. It’s especially interesting because by letting the users execute any binaries we’re increasing the exposition of their systems. Hence, jumping from our distributions’ nest into the lions. As always, sandboxing creates new challenges: It requires changes in applications (or frameworks) to adapt, often creating a user interaction fence (e.g. a popup asking if you let Kamoso access the webcam). For what it’s worth, that’s not new: Android does it, OS X does it, Windows does it (from the Store), Chrome OS does it, etc.

Now where are we?

We need to decide about GNU/Linux’s future. Or at least, we need to understand what Plasma users will have available. So far, most of the noise comes from the big players in the business trying to differentiate their products, meaning incompatible versions.

Without an agreed unified solution, we’ll have to assume we’ll end up having installed snappies, flatpaks, AppImages as well as applications from the distribution. Then it’s just a matter of:

  • Presenting it properly so that the user knows the risks taken by executing an application (!)
  • Make sure we don’t lose many features by sandboxing.

Still, one of the good things of this new approach is that it shouldn’t have to be necessary to have several people dedicated to build every single application and component. If the solution is to add 3 more solutions that will need dedicated people, we’re not really moving forward.

Building

As soon as we’ve decided how we want to work, then the interesting stuff needs to appear. If this is properly engineered, it can bring really interesting possibilities that now we hardly ever find:

  • Newer versions of applications on administered systems (e.g. universities).
  • Enabling stable distributions on professional environments.
  • Beta channels.
  • Binary application 3rd party extensions.
  • Provision of debug symbols (some distros don’t offer them).

To finish the fantastic post, a note for the dreamers:
How easier would all that be in a microkernel architecture?

We need you!

Of course this will be a long journey and we need your collaboration. This year in Randa we started working on all these problems in several different angles. It’s important for the KDE Community to have your support, so we can keep providing quality software. Consider donating, doesn’t need to be a lot, everything counts.

Discover in Plasma 5.7

I haven’t talked about what’s going on in the Discover front for a while, here’s a small heads up on what you’ll get in the next Plasma 5.7 release.

Back Ends

  • PackageKit (and AppStream): This one has received most of the care. It has been rearranged to instant load. Also we made it possible to integrate this backend with KNS/OCS, to show when there’s add-ons ready for an application.
  • KNSBackend: It’s being polished and adapted for new use-cases. First it used to focus on Plasma-centered assets, but we’ve worked towards making it possible to become a viable solution for Application add-ons as well. Furthermore it has been trimmed down to also provide fast startup while keeping its features.
  • QApt: The traditional QApt backend used in Kubuntu is now considered deprecated in favor of PackageKit, which I understand should be available properly on every GNU/Linux distribution, including Kubuntu. We kept it around because there’s been longstanding issues with PackageKit on Ubuntu, but I’m confident these will be solved soon.

Front End

We used to have many of them, they have now been reduced to two: Discover and the Notifier Plasmoid. This allows us to focus one GUI. Of course, you can still use Muon, Apper and Synaptic together with it, for more precise package management.

Regarding Discover, the main focus this year has been: performance, stability and Add-ons.

  • Performance: Mostly back-end work, and it has also benefited from improvements in KF5 and upstream Qt. Updating the dependencies will improve your Discover experience as well as that of other similar applications. Better usage of some model classes was also a nice improvement in some places.
  • Stability: We worked on making sure that things not only are sturdy but that the development process doesn’t get in the way of stability. To that end, I set up a testing framework so that changes are automatically tested, additionally to the manual tests the developers will do at some point.
  • Add-ons: One of the important parts of Discover, is to make sure the user is aware of the nice resources he has available. KNewStuff resources (mostly coming from kde-look and kde-files so far) have been available for a while, albeit mostly for Plasma enhancements. In this next release we allow these to extend applications themselves as well.

Future

Discover‘s future is Kirigami. We have already some work lined up (see the kirigami branch), we are pending some design work so that it shines, but the approach is already quite promising!

Also I’d like to start working on the Flatpak backend, if nobody beats me to it. 😀

Stay tuned!

AppStream offers a Cross-Distribution identifier for applications

One of the problems we’ve historically had in the GNU/Linux world (and especially in KDE, since we’re cross-distribution) is that we’ve always had a hard time at explaining how to install our applications. Basically because it’s different on each distribution.

My first approach to this problem was creating Muon Discover but that’s clearly not enough because, in the end, we seldom look for new applications. Often they will be recommended on a blog post or forum and we’ll want to install it. But then, while recommending it, it will either be a vague “you should try KMail” or a “you should ‘apt-get install kmail’ or whatever applies to your distribution.

What I’m proposing today is to embrace AppStream at the level of URL handlers (for web browser integration). AppStream shows GNU/Linux distributions maturity by offering a solution by letting the upstream projects set the identifier in their appdata.xml file (among other things) and then let distributions name the package however they want, rather keeping to push different naming schemes down each other throats.

AppStream has been supported in Discover for over a year now (only for PackageKit so far) and, since introducing this feature is straightforward, I implemented it in master.
If somebody wants to give it a go, test master and click here: KMail

I think this will be beneficial not only for KDE but any project packaged on GNU/Linux distributions that have a presence on the web and need to find their way in users’ installations. This will of course be limited to getting the URL scheme adopted by our peers.

/me winks at Gnome Software and Ubuntu Software Center.

Plasma Mobile SDK

Where are the giants?

When approaching this issue I had been thinking about the issue for a while. I had mainly 2 problems: I was rather frustrated with previous Linux-based systems so far and the one I liked didn’t really scale for us. One thing was clear: We had to stand on the shoulder of giants.

  • The first one to think about was the N9 SDK (and by extension N900). It used to have a scratchbox-based system that emulated the one on the phone. It was useful for testing the applications locally on the device (although I actually never used that), I think that this one had both cross-compilation toolchain and base system, as well as the host and it used QEmu if a host executable was run. It felt weird because it shoved you in a weird system and you had to pull your code in weird ways to fetch it. Otherwise it worked great. Afterwards, madde came. It’s what I was looking for really and played quite well with cmake, actually when I published the steps on how to develop for the N9 it’s what I used back then, but it probably was too late for it to become a thing I guess. Also many people weren’t too fond of it, as I learned after some time.
  • The Android NDK is the other I took into account. I don’t think it would be fair to compare anything we could do with the actual Android SDK, so I’ll limit myself to this one. This one ships the complete cross-compilation toolchain and runs native (similar to the BBX SDK, IIRC).

For the N9 approach, I would have had to concentrate on figuring out technologies that were long dead (and probably should remain). For the Android approach I found 2 big problems: We have to actually work on generating the actual binaries (which means, from any platform to a specific ubuntu-vivid-arm target) plus all the dependencies. This meant, creating a new distro, and we already had debian/ubuntu for that, let’s use debootstrap! Oh, wait…

Old school

For a start, I took what sebas already started, in fact. Using debootstrap to create a chroot jail that could cross-compile the projects into our platform. This started to prove feasible really soon, as 2 or 3 days after working on it (and after fixing some issues that kept arising, mostly on KF5 itself and packaging), I already started to output binaries that could be deployed on the device.

Problems:

  • My IDE is outside of the jail, so there isn’t much we can do to integrate at all (we can’t access the build directory, and most of the data in it isn’t too meaningful for the outside anyway). A solution would be to ship and run the IDE from within the SDK though.
  • You need to be root. Not only to generate the system, but also to run it.

New school

An idea I wanted to approach was docker. Everyone on the web world is shit-crazy about it and it’s deeply based on traditional Linux, so there’s quite in common already. Doing something similar to the deboostrap there was a piece of cake, and I managed to re-use the set up code I already had for the previous version.

Problems:

  • Also everything is inaccessible from the IDE
  • I’m less aware of the limitations I’ll find.

Still, this second approach feels lighter and it’s quite fun to investigate something new like that.

Tooling

It seems that the jailed systems are the way to go, at least for now, so the tools I’ve created so far assume that they live in the jail as well. So, what do we have?

  • createpkg: Makes it possible to create a deb package that can be sent to the device for testing. It’s much simpler than the simplest deb package ever, but it works. Much better than sending executables over, much better than learning how to package for Debian.
  • deploypkg: sends the deb file, installs it and starts the application.
  • click-get: Downloads a click file from the Ubuntu Store, it’s harder than you’d think.
  • kdev-debdeploy: Does it all, inside your good ol’ IDE.

To be done

  • Workflow: Figure out a way to deploy KDevelop within either jail, but then we’ll also be able to use kde-debdeploy, that does mostly the same as these tools, integrated on the IDE.
  • Emulation: Testing: QEmu + docker anyone?

Suggesting new ways: Kamoso 3.0 Technology Preview

The world changes, and with it, we change too. For this new version of Kamoso we wanted to iterate what we’re presenting.

The Camera

Since we’re using GStreamer, we could take advantage of newer technology available there. Now we’re using camerabin for the video capture (in contrast to using v4l2src directly).
This will offer a better experience, with less tweaks on our side, that will definitely improve the end-user experience.

User Interaction

The whole application wasn’t rewritten, but all the UI was. We have a re-designed interface that integrates some really lean and meaningful animations and sorts the components so that we can get a simple, dialog-free UI.

Sharing

Sharing is probably the most important feature in Kamoso besides video capture, and we took special care of it. We adopted the new Purpose technology that is seeing its light for the first time with this Technology Preview. We want to make sure that we can offer the best solution by making sure it integrates properly on our UI and it can also be integrated on other parts of KDE. This way we’ll be able to come up with an adequate set of plugins and enjoy them all in community. For now, we have: KDE Connect, Imgur for images and YouTube for videos.

Where to find it?

I put together this little list, so everyone can find the tarballs and packages (on the distributions that managed to package it):
https://userbase.kde.org/Kamoso/3.0#Alpha

We need you!

We need special help with:

  • Testing, on more cameras, more hardware.
  • Artwork, especially icons.
  • Coding, especially Purpose sharing plugins (Facebook? Twitter? ownCloud?)

But anything else is welcome, as always. Please get in touch!

Identities, I usually don’t stop being myself

One of the most interesting developments I’ve seen recently inside KDE is KAccounts (or Web Accounts, as it used to be called). It’s not even a KDE project, but a project Nokia started some years ago, I’m guessing that on MeeGo days.

Why do I care?

As a developer it’s always very important to know what resources you have available. The more you have, the richer the users’ experience will be. Resources are not only hardware but often just semantically-structured information. For example, KPeople helps us tell apart the contact list and offers us people instead or Solid, that turns the different ways of exposing the hardware information into something easily consumable by front-end applications.

Now we’re talking about identities. We don’t exclusively rely on local data anymore, I think that’s a fact nowadays (and even 5 years ago). Our disks are just another place where we store some of our data. KDE projects must be prepared to easily offer such workflows. One of the important parts to do it properly is authentication:

  • We need a list of available services we can use.
  • We don’t want our users requested identity authentication all the time.
  • We don’t want each application to implement the N authentications algorithms of the N different providers our users will eventually have.

Why do you care? (as a KDE software user)

First of all, you want your developers to have the best tools (GOTO: “Why do I care?”).

There’s a more important issue to figure out, more ideological than technical: We ask service and software providers to use standard specifications to improve interoperability, but then in practice we are seldom leveraging it. The case-by-case set up burden makes it hard for users to have everything configured properly. An example for this is ownCloud integration. I know a couple of people running an instance, but they seldom integrate it in Dolphin, but even if they did they’d still need to integrate the rest of provided services, such as calendar, contacts, etc.
We want to tell our software what providers we have available, those providers will provide different services. The software should integrate as much as possible with those services.

How can I help?

  • Testing: There’s a lot going on, thorough testing and ideas is important. Both when it comes to making sure everything works as expected as well as what software should adopt KAccounts. It will already be available with KDE Applications 15.04.
  • Developing: Make sure your application of choice uses KAccounts to figure out the configured services. Here’s a good place to find documentation: API, and an example use-case.
  • Promotion: It’s important to reach out to service providers, make sure they know there’s such possibility and hopefully they can help us work better with their services (hey! by the book example of a win-win situation!).

Nowadays our computation experience is spread among different services and we need to be smart enough to understand how our users will adopt them and make it as transparent as possible.
AccountsSSO and KAccounts are a solid step forward in this direction, let’s get back in control of our data!

A Purpose for everything

When we were porting Kamoso to Qt5/KF5, at some point I realized that it was about time we came up with whatever we’d want to do with sharing. Kipi is definitely an interesting technology, but no matter how I looked at it I found that it missed an iteration in the concept. In some aspects it’s very specific, in some others very broad. In fact, I already tried to improve it, back in 2009.
My conclusion was that I wanted to take a step back and re-think what I wanted because I don’t think it’s just about images and it’s not even just about Importing and Exporting.

  • Is Import/Export really special?
  • What do we want to integrate?
  • How can we simplify the implementation as much as possible? Having as many plug-ins as possible is a must.
  • What are other platforms doing?

This took me a while to figure out and it’s probably not ready yet, but since I needed something anyway I decided to just go for it.

What?

The idea is that at some point when developing an application you’ll need to fulfill different purposes (or desires, as some like calling it). At this point we’ll want to integrate the possibility of providing an action with all the information needed to fulfill it.

This purpose will have different fulfillment propositions, which we’ll need to provide a list with the alternatives and then be able to go for the one the user prefers.

Example 1

In Gwenview we want to be able to share the currently displayed image. To do so, we’ll get a toolbar button that on click we’ll get the 2 more likely alternatives and a “More…” option that will open a dialog that lists all the possibilities.

With all the provided information, we should be able to perform the export with minimal and integrated configuration. The different alternatives could be exporting to imgur, saving the file locally, upload to my ownCloud or my Facebook. For this, note that any configuration will seldom be needed.

Example 2

We’re reviewing code in KDevelop and we want to tell on someone we messed up. KDevelop can integrate the “Call” purpose that will propose us all the different ways to get in touch of the disaster perpetrator.

Example 3

The QuickShare plasmoid has a patch file dropped on. We receive the alternatives which can have any mime type: text files, binary files and any specific format (such as text/x-patch or text/markdown).

Example 4

An application wants to show us a location in a map. Instead forcing into some implementation, it will request Purpose to show a latitude and longitude and the user will decide where to check it. Alternatives could be OpenStreet Map, Google Maps and Marble.

How?

I came up with a small proof of concept to be able to start getting things working and I even adopted it on a couple of projects. It’s working well but I’m still a bit afraid of committing to an API. As a first step, I’m here explaining it for you all.

At the moment the API is really simple, it consists of 3 C++ classes:

  • AlternativesModel, the interface for clients.
  • PluginBase to be implemented by each plugin.
  • Job which will be re-implemented by the plugin and tracked by the client

Then there’s some QtQuick components built on top that simplify the integration on QML applications. We should get some QtWidgets components as well, as soon as we have to integrate it on a QtWidgets-based application.

The AlternativesModel will receive a PluginType string which defines what kind of plugin will be used and the input data. The plugin type will define what information needs to be provided and what output will be received. Then the model will be populated with all the plugins with the said PluginType. The plugins will be able to specify a set of constraints, so the model can filter them out in case they’re not a good fit, and a set of configuration variables. These configuration variables are required to run the plugin and if they’re not provided, the application will need to show the user interface provided by the plugin, which is implemented through a QtQuick file, for now.

For some more precise information, here’s

TextField { id: url }

Purpose.AlternativesView
{
  pluginType: "Export"
  inputData: {
    mimeType: 'text/plain',
    urls: [url.text]
  }

  onFinished: {
    console.log("Done", output.url)
    if (error!=0) {
      console.log("error:", errorString)
    }
  }
}

Where are we?

It can be used now. It’s already integrated in the QuickShare plasmoid in kdeplasma-addons and in Kamoso which should be released eventually. Still, I’d like to see other use-cases being implemented and developed. The biggest void I see is plugins: getting Google+, Twitter, Facebook, ownCloud should be easy but needs to be done. If somebody is knowledgeable about such API’s and is interested in KDE applications integrating them, please step forward!

Additionally, if you think your application could use some purposes or that you have an idea of a new purpose that will change humanity, don’t hesitate to get in touch. We’ll change it together! \o/

Now here’s a screencast of the work in progress (especially the UI, eh..)

PS: Video uploaded using Purpose as well!

Communicating from Plasma 5

Porting KDE Telepathy to Qt 5 and Plasma 5

I started working on that port back in the last KDE Telepathy sprint in Barcelona last April. Back then, I started to work on it because I have been doing heavy usage of the KTp plasmoids back when using the KDE 4 Plasma series and I didn’t want to live without them. Back then, I only ported the minimum parts of ktp-common-internals so it would work with KF5, as well as the plasmoids. It was quite some work, but definitely worth it since I’ve been using them ever since, and it’s worked wonderfully.

Last week I started working on those ports again, this time trying to start get all of them ready for end-users, first step being starting to port the rest of modules. It’s worth mentioning how good the response has been, given that many people chipped in and gave some modules a go. It’s a bit weird to do this kind of porting in KTp, because there’s tons of little repositories to port rather than a big one, but I guess it’s kind of part of it’s beauty anyway… 😉

KPeople as a KDE Framework

KPeople is a Framework for fetching contacts from different sources (Telepathy, Akonadi, Facebook, etc) and unifying them into a same model,

An important part of making sure all of KTp works is ensuring that its dependencies are up to speed and this time the one I’d like to bring some light to is KPeople. The port is ready really, only depending on having some of its own dependencies from kdepimlibs in a releasable state, but it’s also quite in shape too. It’s a framework I’d really like to see shining in the next months.

Furthermore, I finally managed to find some time and get the automatic contact merging back on. This I started more than a year back and then Franck Arrecot worked to make some GUI interface for it, I think it’s quite interesting. Take a look into it if you think it’s interesting. 🙂

We need you!

Last but not least, there’s still lots to be done. I’d like to aim for a nice and clean release of KTp by the end of the year, ready to be shipped with Plasma 5.3 and the applications, if the maintainers allow so.

So if you’d like to help, you can take a look at this Kanban board we created and take the tasks you’d like.

Taking advantage of OpenGL from Plasma

I’m excited, and I hope you’ll be too.

David Edmundson and I have been working hard the last weeks. It’s not that we don’t usually work hard, but this time I’m really excited about it.

A bit of context: in Plasma an important part of the system drawing is painting frames (others are icons, images and the like). Those are in general the elements that are specified in the Plasma themes. These will be buttons, dialog backgrounds, line edit decorations, etc.

So far, to paint those we were assembling the full image in the CPU and then sending it over to OpenGL as a full texture, then we would do the composing of all the different frames, according to the information provided by QtQuick, through the Qt Scene Graph. There are 2 main problems in the current approach.

  1. We were maintaining huge textures in memory. Every frame was completely stored in memory and gpu memory. Which means that the bigger the dialogs are, the more memory we consume, even though the texture is flat.
  2. Every time we resize the frame, we have to re-assemble the frame in CPU memory and upload it again.

First: The 9-patch approach

First we made it possible to have the frames to be rendered by each different parts and assembled by the GPU. This wasn’t possible, because Plasma themes are quite complex, so now we have 2 different paths. If the theme element can take advantage of the optimization, it will use the new code, otherwise it will stay working beautifully on the former, thorough implementation [1].

Therefore, instead of rendering all the frame now we’ll be uploading 9 textures to the GPU, and let it either tile or stretch depending on the settings in the theme. This way:

  1. we are uploading 9 tiny textures rather than a big texture.
  2. when the frame is resized, we tell the nodes to resize and the GPU does the job [2].

Second: Caching the textures

Now everything was in place, we’d have many times the small 9 elements but we kept uploading them to the GPU over and over. It’s little textures, but it’s still better if we get to re-use what we already have. To do so, we’ve placed a little hash table that keeps track of the already created textures to re-use them. This way, we get to tell the Qt Scene Graph to use a texture that has already been uploaded rather than a new one. We’ve run some tests, here’s the result:

  • In PlasmaShell we get 342 miss and 126 hits, so roughly 25% of bandwidth and memory improvement
  • In KRunner we get 108 miss and 369 hits, so roughly 350% improvement on memory and bandwidth improvement.

Future, further work

Sadly enough, raw memory usage is still quite high, when running plasma shell on massif, we are still reported as most of the memory usage being in the GPU graphics card (or rather i965_dri.so), so we’ll have to dig it [3]. We’ve found some ways to improve this, for example by enforcing OpenGLES 2, but this requires Qt 5.4 which is due October. I implemented it nevertheless, and it works fine.

Being more precise, a big offender is using a wallpaper image. We’ve looked into it, the code looks fine, but then it makes a big difference, so big that I still don’t understand how it can be. A good suggestion if you’re testing Plasma 5 on a system low on memory, is to run it with the plain color wallpaper. We can save up to 30% of memory consumption, no kidding (it actually depends on who you ask, either massif, htop or ksysguard; but they all agree it’s a big deal). We’ve investigated a bit and found ways to improve the situation there, but if you are interested, feel free to join!

Finally, another problem with regards to memory consumption is QML. We make heavy usage of it and it shows memory-wise. We should see if we can adopt any optimization to stream-line our usage, but admittedly it’s much better than one would have expected.

Testing

If you want to give it a try, you can already find most of this in master, and it will be available from the next KDE Frameworks 5.1 release which will be available by the second week of August.

Hope you liked it, it was a great exercise to investigate all this! I learned a lot and gained quite some respect for the Qt Scene Graph and QML development teams. Keep rocking!

[1] More precisely, at the day, when there’s no hint-compose-over-border or mask-*-overlay elements

[2] an exception for it being (hint-stretch-borders and hint-tile-center hints, where we’ll have to re-render on resize it).

[3] David, Vishesh and I we all have Intel drivers, but I guess it’s a good card to test-case on, given how mainstream it is, currently.